I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing
Authors: Yiwei Ma
Jiayi Ji
Ke Ye
Weihuang Lin
zhibin wang
Yonghan Zheng
Qiang Zhou
Xiaoshuai Sun
Rongrong Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: Benchmark
Instruction-based Image Editing
Multimodal Large Language Model
Primary area: evaluation

Abstract: Significant progress has been made in the field of Instruction-based Image Editing (IIE). However, evaluating these models poses a significant challenge. A crucial requirement in this field is the establishment of a comprehensive evaluation benchmark for accurately assessing editing results and providing valuable insights for its further development. In response to this need, we propose I2EBench, a comprehensive benchmark designed to automatically evaluate the quality of edited images produced by IIE models from multiple dimensions. I2EBench consists of 2,000+ images for editing, along with 4,000+ corresponding original and diverse instructions. It offers three distinctive characteristics: 1) Comprehensive Evaluation Dimensions: I2EBench comprises 16 evaluation dimensions that cover both high-level and low-level aspects, providing a comprehensive assessment of each IIE model. 2) Human Perception Alignment: To ensure the alignment of our benchmark with human perception, we conducted an extensive user study for each evaluation dimension. 3) Valuable Research Insights: By analyzing the advantages and disadvantages of existing IIE models across the 16 dimensions, we offer valuable research insights to guide future development in the field. We will open-source I2EBench, including all instructions, input images, human annotations, edited images from all evaluated methods, and a simple script for evaluating the results from new IIE models. The code, dataset, and generated images from all IIE models are provided in GitHub: https://github.com/cocoshe/I2EBench.

E'Beench A Comprehensive Benchmark for Instruction-based Image Editing
Weihuang Lin', Zhibin Wang?t Xiaoshuai Sun't Rongrong Ji' Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Yiwei Ma
Jiayi Ji Ke Ye! Yonghan Zheng' Qiang Zhou²
2 Inf Tech Company, Hangzhou, 310000, P.R. China. yiweima@stu, xmu edu. cn xssun@xmu.edi edu.
Abstract
Significant progress has been made in the field of Instruction-baset Image Editing (IIE). However, evaluating these models poses a significant challenge. A cru- cial requirement in this field is the establishment of a comprehensive evaluation benchmark for accurately assessing editing results and providing valuable insights for its further development. In response this need, propose EeBBench, comprehensive benchmark designed to automatically evaluate the quality of edited images produced by IIE models from multiple dimensions. I'EBench consists of 2,000+ images for editing, along with 4,000+ corresponding original and di- instructions. It offers three distinetive characteristics: 1) Comprehensive Evaluation Dimensions: FBBBnch comprises 16 evaluation dimensions that cover both high-level and low-level aspects, providing comprehensive assessment of each IIE model. 2) Human Perception Aligument: To ensure the alignment of our benchmark with human perception, we conducted an extensive user study for each evaluation dimension. 3) Valuable Research Insights: By analyzing the advantages and disadvantages of existing IIE models across the 16 dimensions, we offer valuable research insights to guide future development in the field. We will open-source E'EBehhh, including all instructions, input images, human annotations, edited images from all evaluated methods, and simple script for evaluating the results from new IIE models. The code, dataset and generated images from all IIF models are provided in github: https: /et:://ttttte gigh........ com/ cow.s/cccccccccccoooooocoo.cc IIee/eeeBBee..
Introduction
instruction-based Image Editing (roo)s et al. [2023], Geng et al. [2023], Zhang et al. [2024a]] Li al. [2023c], Wang et al. [2023b], Zhang et al. [2023a], et [2024], which aims edit an image using text instruction, provides user-friendly way for the community to edit images. Over the past few years, significant progress has been made in IIE, with the development of diffusion models Ho et al. [2020], Sohl-Dickstein et al. [2015], Welling and Teh [201 11, Kulikov et al. [2023] and large vision-language models (LVLMs) Liu al. [2023a,b], Fei et al. [2024c,a,b], Ma al. [2024]. However, there is a pressing need for comprehensive benchmark to effectively assess the performance of these models. An ideal evaluation framework should not only measure the editing quality across different dimensions but also align with human perception to ensure reliable measurements. Furthermore, the evaluation should highlight the specific strengths and weaknesses of each model, thereby offering valuable insights for future endeavors in data selection, training
"Equal contribution. t Corresponniig author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Human Annotation
Automated Evaluation
Human Evaluation
IEE Model
Content Categoties
Exaleation
Editing Types
2000 Images
High-LLve Editing
Lav-Leve Editing
2000+
Original Instructions
2000+ Diverse Instructions
Couneing
Deblerring
Questionnaire
Dircction Perceptiion
Hore Remsvval
Image Editing
Objeett Removal
Lowlight Knhancement
Alignment Verification
Object Replacement
Neise Remaval
Background Replacement
Kain Removal
Comparison
Coler Altoration
Shadon Removal
Analysi sis
IIE
Style Alter attionaioon
Snow Renoval
Human Preference
1 Ehench Score
Model
Made
Imput
Output
Accuracy
Watermark Remmnal
Overview of 1'Beenhh an automated system for evaluating the quality of editing results generated by iustruction-based image editing (II)) models. We collected dataset of over 2000+ images from public datasets Lin et al. [2014], Guo et al. [2023b], Martin et al. [2001], Chen et al. [2021], Ancuti et al. [2019], Liu et al. [2021b,a], Qu et al. [2017], Nah et al. [2017]. Shen et al. [2019], Wei et al. [2018] and annotated them with corresponding original editing instructions. To diversify the instructions, we used ChatGPT Achiam et al. [2023] to generate varied versions. With the collected images and the original/diverse editing instructions, we utilized existing IIE models to generate edited images. Subsequently, we developed an evaluation methodology to automatically assess the adherence of edited images to the provided instructions under different dimensions. We also implemented human evaluation to obtain human preferences for editing results of different IIE models. Finally, we analyzed the correlation between automated evaluation and human evaluation, confirming alignment with human perception.
Figure 1:
strategy selection, and architecture design within this field. However, evaluating an IIE model poses challenges due the diverse range of editing types and the inherent difficulty assessing the level of alignment between edited images and given instructions.
Existing evaluation metrics for IIE could be divided into three categories: 1) conventional metrie; 2) user study; 3) benchmark. The first category Brooks et al. [2023]. Geng et al. [2023], Zhang et al. [2024a], Li et al. [2023c], Wang et al. [2023b], Huang al. [2024b] employs conventional metrics to evaluate IEE models, including CLIP Score Radford et al. [2021] ]. CLIP Text-Image Direction Similarity Radford et al. [2021], PSNR Korhonen and You [2012]. SSIM Wang et al. [2004], and LPIPS Zhang et al. 22018]. The ad advattage of this approach is its case of use. However, a single metric not suitable for evaluating all types of editing. For instance, CLIP score measures the similarity between images and text, making it less suitable for low-level visual editing tasks like denoising and low-light enhancement. Similarly, PSNR, which measures image similarity, not adequate for high-level visual editing tasks such as object removal and replacement. The second category Li et al. [2023c], Zhang et al. [2023a], Fu al. [2024] involves methods that evaluate the effectiveness of different techniques by soliciting ratings from human participants. This approach directly reflects human preferences and aligns the results with human perception. However, it is costly method and lacks reproducibility. as the test sets and participants may be not consistent in each evaluation. The final category comprises benchmarks Kawar et al. [2023]. Wang et al. [2023c], Basu et al. [2023], Huang et al. [2024a] specifically designed for evaluating IIE models. While these benchmarks are tailored for IIE, they have certain limitations. For example, TedBench Kawar et al. [2023] evaluates only 100 images with commonly occurring editing types, which may not sufficiently demonstrate the capabilities of IIE models. EditBench Wang et al. [2023c] focuses on mask guiik- editing, rendering it unsuitable for evaluating mask-free methods. In Edit Basu et al. [2023], only limited set of dimensions related to size or location can be automatically evaluated, limiting its universality.
In this paper, we propose IBBBench, a comprehensive benchmark designed to automatically evaluate the performance of IE models. I'EBench exhibits three attractive characteristics: 1) Comprehensive Evaluation Dimension, 2) Human Perception Alignment, and 3) Valuable Research Insights

---

Original
Any2Pix
Hive
iEdit
Instruct- diffusion
Pix2Pix
Magic Brush
Mgie
HQEdit
Hightleeel Editing
Background Replacement "Change the background of this photo jungle*
.*....
Object Removai: 'Remove the man from the image"
Low Low-level Editiing
Haze Removal: 'Remove the haze from the imaaee
......
Lowlight Enhancement: "Enhance the brightness of an Image"
Figure Visualization of the editing results on the proposed 16 evaluation dimensions using different IIE models, including InstructAny2Pix Li et al. [2023c], HIVE Zhang et al. [2023a], InstructEdit Wang et al. [2023b], InstruetDiffusion Geng et al. [2023]. InstructPix2Pix Brooks et al. [2023], MagicBrush Zhang et al. [2024a], MGIE Fu et al. [2024], and HQEdit Hui et al. [2024]. A detailed version can found supplementary materials.
First and foremost, E'EBench offers comprehensive evaluation dimension. These dimensions are categorized into two main types: High-level Editing and Low-level Editing. High-level editing primarily focuses on understanding instructions editing specific areas of images, whereas low- level editing more concerned with editing image details or entire image. As shown in Fig. 1, both high-level and low-level editing consist of 8 fine-grained editing dimensions, which serve to demonstrate the model's proficiency in high-level and low-level editing. We meticulously collected approximately 140 images for each editing dimension and annotated each image with original editing text instruction. To diversify the instructions, we also utilized ChatGPT Achiam et al. [2023] enhance the description of the instructions and obtain wider range of variations. In addition the multi-dimensional evaluation, we also conducted multi- multi-category evaluation to assess the model's performance on different content categories. To achieve this, we included additional annotations for each instruction with different categories such as Animal, Object, Scenery, Plant, Human, and Global.
Second, the FBBenhh score aligns with human perception. This is accomplished by collecting scores from human annotators for the outputs generated by different IIE models, covering multiple evaluation dimensions. By conducting comprehensive analysis of both the E'BBench scores and the human scores, we have identified a substantial correlation between them. This discovery serves compelling evidence, affirming that proposed evaluation approach closely aligns with human perception.
Lastly, EBBech offers valuable research insights through its systematic evaluation across various dimensions and categories. The proposed J'EBench not only facilitates a comprehensive assessment of existing models but also derives valuable insights into their respective strengths and weaknesses. These insights act roadmap for enhaneing architecture design, refining data selection strategies, and ultimately elevating the quality of editing outcomes.

---

We are open-soureing EBEBenhh, including all instructions, input images, human annotations, edited images from all evaluated methods (like Fig. 2). and a simple script for evaluating the results of new IEE models. By making these resources freely available, we aim to foster fair comparisons within the field and facilitate valuable insights for community development.
2 Related Work
2.1 Instruction-based Image Editing
With the advancements in Generative Adversarial Networks (GAN)Goodfellow al. [2014, 2020], Mao et al. [2017], Karras et al. [2019], Yoon et al. [2019], Karras et al. [2020a], Chen et al. [2018], Zhang et al. [2019] and Diffusion modelsSong et al. [2020], Ho et al. [2020], Nichol and Dhariwal [2021], Kawar et al. [2022], Austin et al. [2021], Dockhorn et al. [2022], text-to-image models Saharia et al. [2022], Rombach et al. [2022], Ramesh al. [2021, 2022], Betker et al. [2023]. Karras et al.
[2019, 2020b] have made remarkable progress in recent years. As the demand for image editing continues to grow, a multitude of text-based image editing Xu et al. [2024], Kawar et al. [2023], Zhang et al. [2023b], Saund et al. [2003]] Zhang et al. [2024b] models have emerged. One editing task, known as Prompt- based Image Editing (PIE) Avrahami et al. [2022], Valevski et al. [2023], Hertz et al. [2022], Dong et al. [2023], requires users to provide target description along with the original image. The PIE model then analyzes the target description to modify the input image accordingly, generating target image that matches the provided description. However, despite the lowered threshold for image editing, the requirement of describing the entire content of the target image in the description still poses challenges in terms of user interaction. To address this limitation, Instruction-barn based Image Editing (IIE) Brooks et al. [2023]. Geng et al. [2023]. Li et al. [2023c], Fu et al. [2024], Huang et al. [2024b] was proposed, which simplifies the user's s role to providing the original image and modification instructions (e.g.. 'Remove the dog'). ) One notable implementation, InstructPix2Pix Brooks et al. [2023], introduces large-scale dataset for instruction-bate editing. The dataset is created using a fine-tuned GPT-3 Brown et al. [2020] and image pairs generated by the Cormnpttt--rrompp diffusion model Hertz et al. [2022]. Additionally, InstructPix2Pix proposes an instruction-basse diffusion model for image editing based on this dataset. However, due to the automatic generation and filtering of the InstructPix2Pix dataset, concerns arise regarding its quality and potential noise. To address this, MagicBrush Zhang et al. [2024a] proposes a manually-annotual eestruction-guider image editing dataset. In addition textual instructions, InstructAny2Pix Li et al. [2023c] proposes a model that utilizes other modalities, such as audio and image, as instructions. To enhance the level of detail in instructions and improve the accuracy of editing results, MGIE Fu et al. [2024] introduces the use of Multimodal Large Language Models (MLLM) Liu et al. [2023a]. SmartEdit Achiam et al. [2023]. aiming to improve the editing capabilities of IIE models in complex scenes, incorporates MLLM into the IIE model to better comprehend instructions. Despite significant progress, evaluating the editing performance of HIE models remains a crucial concern. Therefore, in this paper, we present 1eBBench, a systematic evaluation framework for these models. Our work includes an in-depth analysis of their strengths and weaknesses, offering valuable insights for the future development of IIE models.
image
2.2 Text-based Image Editing Benchmark
While numerous benchmarks Marino et al. [2019], Hudson and Manning [2019], Bigham et al. [2010], La et al. [2022], Li et al. [2023d,a,b], Yu et al. [2023], Wu et [2023b] have been introduced for evaluating vasion-language tasks Wang al. [2024b,a, 2022], Wu et al. [2023a], Dai et al. [2024], Hu et al. [2024], the evaluation of text-based image editing models often relies on metrics such as CLIP Score Radford et al. [2021]. PSNR Korhonen and You [2012], SSIM Wang et al. [2004], and LPIPS Zhang et al. [2018]. Several existing studies have introduced benchmarks to assess the performance of image editing models. TedBench Kawar et al. [2023] presents a relatively small benchmark consisting of only 100 images and a limited set of highly common editing types. EditBench Wang et al. [2023c] is specifically designed to evaluate mask-guided image editing methods, which necessitate the availability of additional masks indicating the areas to be edited. In Edit Basu et al. [2023], the evaluation of certain dimensions relies on manual labor, thereby limiting the reproducibility of performance. Moreover, the remaining dimensions primarily involve modifications to object size or position, lacking comprehensive coverage. While MagicBrush Zhang et al. [2024a] and Emu Edit Sheynin et al. [2023] propose test sets for evaluating editing performance,
4

---

they still rely on conventional metrics such as LI, L2, CLIP-I. DINO, and CLIP-T, which may not accurately capture the nuances of all editing types. SmartEdit Huang al. [2024b] specifically develops benchmark tailored for complex editing scenarios, but it does not accommodate other editing scenarios. Considering the current absence of systematic benchmark that comprehensively evaluates the editing performance of IIE models across different editing types, we propose EBEBench to address this gap.
3 EEBeenh
This section provides overview of the main components of E'EBench. In Sec. 3.1, we provide concise introduction to the principles, definitions, and evaluation methods of 16 dimensions. Sec. 3.2 outlines the process of data annotation. Lastly, in Sec. 3.3, we present the human evaluation process to assess the correlation between the E'EBench score and the human score. A detailed explanation can be found in the supplementary materials.
3.1 Evaluation Dimension
In our evaluation of the IIE model's editing quality, we have categorized it into 16 dimensions, each assessing different aspects of editing in a top-down manner. An overview of FBBench is presented in Fig. High-level Editing Evaluation primarily focuses on assessing the model's ability accurately understand instructions and make precise edits to local areas of the input image. This evaluation consists of 8 dimensions. Low-level Editing Evaluation, on the other hand, primarily examines global editing and detailed image processing. It also comprises 8 evaluation dimensions. Unlike previous approaches Fu al. [2024], Zhang et al. [2023a], Geng et al. [2023] that relied on single metric, such as CLIP score Radford et al. [2021]. to evaluate editing quality for all editing types, we have developed specialized evaluation methods for each of the 16 dimensions. This approach is necessary due to the distinct goals of high-level and low-level editing.
3.1.1 High-level Editing
Evaluating editing quality in high-level dimensions poses challenge due to the diverse goals involved, making it impractical to rely on a single metric. The advancement of Multimodal Large Language Models (MLLM) Gao al. [2024], Chu et al. [2024], Zhu et al. [2024], Dong et al. [2024], Ma et al. [2022, 2023]. Ji et al. [2022], such as GPT-4V Achiam et al. [2023], Gemini Pro Reid et al. [2024], and QWen VL-PPlus Bai et al. [2023], has significantly enhanced automated understanding of images. Therefore, to ensure precise evaluation of the editing quality of IIE models in high-level dimensions, we leverage the exceptional capabilities of the widely recognized GPT-4V model make judgments for most high-level evaluation dimensions.
Counting.
The Counting dimension pertains to instructions related the number of objects, such as "add two apples to the image." " To assess this dimension, we query GPT-4V about the number of target objects in the image and compare its response with the human-annotated answer.
Direction Perception. The Direction Perception dimension requires the IIE model to comprehend directions provided in instructions, and accurately make edits when presented with images. We evaluate this dimension by asking GPT-4V if the target object is in the expected position.
Object Removal. The Object Removal dimension focuses removing the target object according to the given instruction. To evaluate this dimension, we inquire whether GPT-4V identifies the presence of the target object in the image.
Object Replacement. The Object Replacement dimension aims to replace the original object with the target object as instructed. To assess this dimension, we query GPT-4V about the presence of the target object the image.
Background Replacement. The Background Replacement dimension involves replacing the original background with the target background as specified in the instruction. To evaluate this dimension, ask GPT-4V if the background of the image matches the textual instruction.
Color Alteration. In the Color Alteration dimension, we modify the color of the target object using instructions. To evaluate this dimension, we inquire GPT-4V about the color of the target object in the edited image.
5

---

styley
rreplace mai oo
black
abBle
Rémove
laft
Swapp
cat
ba background
Marty
eemviigg
Numan of
5 5 8.8.8.8.
maze
shadow
bey
was left noiseu
Enhance
Replace
Should
Sater
dog
Jurriness
rainn
watermark
Change
gainting
Blackshoww
plemente
white
Spainting
snow
Wert
brightness
oyberpink
cutratte
Remove
white
Blue
irainoise
scene showy a eEtalude
Beik
corection
Eliminate EnsurePlace
watercolor
(a) Word cloud of original Instructions
(b) Word doud of diverse instructions
(c) Number of images per dimension
Figure 3: Word cloud visualization (a.b) and image quantity statistics (c) of EBBBechh.
Style Alteration. The Style Alteration dimension focuses on changing the style of the image. To evaluate this dimension, we calculate the CLIP similarity Radford et al. [2021] between the edited image and "an image with ... style".
Region Accuracy. In the editing task, we not only assess whether the target area has been edited correctly but also whether areas that should not be edited have been altered. To evaluate this dimension, we sample input images and instructions from the Object Removal, Object Replacement, and Color Alteration dimensions. We annotate the mask for the area that requires editing. Next, we fill the mask area of the images before and after editing with white and calculate SSIM Wang et al. [2004] to evaluate this dimension.
3.1.2 Low-level Editing
Unlike high-level editing, low-level editing instructions are simpler, lacking specifications regarding object size, orientation, or color. Various low-level editing tasks Wang et al. [2023a], Chen et al [2023a], Sanghvi et al. [2023], Chen et al. [2023b], Wu et al. [2023c], Guo et al. [2023a], Kong al. [2022] have undergone extensive development over the years, resulting in a relatively mature evaluation system. Therefore, for low-level editing, we employ the widely recognized metric, namely SSIM Wang et al. [2004], to evaluate the editing quality.
Deblurring, Deblurring encompasses the procedure of mitigating or eliminating blur from images, resulting in enhanced clarity and sharpness.
Haze Removal. Haze removal entails the climination or reduction of atmospheric haze or fog from images, augmenting visibility and reinstating the true colors and intricate details of the scene.
Lowlight Enhancement. Lowlight enhancement refers to the process of improving the quality of images captured in low-light conditions, enhaneing brightness, and reducing noise.
Noise Removal. Noise removal involves the reduction or elimination unwanted noises in images, resulting in cleaner and more visually appealing visuals.
Rain Removal. Rain removal aims to eliminate or reduce the visual effects of raindrops or rain streaks from images, improving clarity and restoring the original appearance.
Shadow Removal. Shadow removal refers to reducing eliminating unwanted shadows from images, enhancing visibility, and improving overall image quality.
Snow Removal The goal of Snow Removal is to effectively reduce or eliminate snow from images.
Watermark Removal. Watermark removal involves the removal or elimination of embedded watermarks from images, restoring the original appearance without the presence of the watermark.
3.2 Human Annotation
Data Annotation. We meticulously curated approximately 140 images from publicly available datasets Lin et al. [2014], Guo et al. [2023b], Martin et al. [2001], Chen et al. [2021], Ancuti et al. [2019], Liu et al. [2021b,a], Qu et al. [2017], Nah et al. [2017], Shen et al. [2019], Wei et al. [2018] for each evaluation dimension of FBBenhh The distribution of the image count for each dimension illustrated in Fig. (c). These images were then meticulously annotated with textual editing
instructions by human annotators, namely original instructions. However, instructions provided by

---

Accutsay
Regien Oobkriing
Haze Ramoval
Reglot Debluning Acturnty
Haze Remool
Attradon
Erhancemen
Style Afsorton
Frhannamert
Lowight
Coler
Color Atteratio
HIVE
Ramional
Noksa
InstructDiffuslon
InstructPix2Pix
Background moomen
Rain
Baskground Replacement
Rain
MagicBrush
MGIE
InstructEdit
Cieeet Rap sosment
Strddow Removal
Objeat Replacement
Stadon
InstructAny2Plx
HQ-Edit
Object
Snow
Neraral
Direction
Countigg
mark
(a)
Birection Peeception
Counting
Waterna's
(b)
Figure 4: Comparison of radar charts for E2EBench scores in different dimensions using (a) original instructions and (b) diverse instructions.
human annotators usually followed a singular sentence pattern. For instance, the prevalent sentence pattern for the object removal dimension typically "remove from the image". To foster increased diversity, we employed ChatGPT Achiam et al. [2023] to effectively rewrite the original instructions. Fig. 3 (a) and (b) present the word cloud visualizations of the original and diverse instructions, respectively. Additionally, we also annotate a category for each instruction, such as animal, object, scenery, plant, human, and global.
Evaluation Annotation. The evaluation process for EBBehhc encompasses two distinet categories. The first category employs conventional metrics assess various dimensions. For style alteration dimension, we utilize the CLIP score as a standard metric, which doesn't 't need any additional evaluation annotations. In the second category, we utilize GPT-4V to evaluate the quality of editing. To facilitate this evaluation, we enlisted the expertise human annotators to annotate questions specifically designed for GPT-4V, along with corresponding standard answers. For instance, let's consider the counting dimension and the instruction "Add a cat to the shoe rack". In this particular case, the annotated question provided by the human annotators is "How many cats are there on the shoe rack?", and the corresponding annotated answer is "One".
3.3 Human Evaluation
The primary objective of the human evaluation is to ascertain the correlation between human percep- tion and the E'BBench score. To achieve this, we present human evaluators with a textual instruction T. an input image V,, and set of edited images [Vi, V.,... VM generated by M different IIE models. The evaluators are then tasked with ranking the results based on their judgment. More specif- ically, we sample N images for each evaluation dimension, leading comprehensive collection of x 16 x 2 edited image comparisons. Within each comparison, evaluators are presented with M edited images assess and rank in relation to another. We assign human model based on its ranking among the M models. Specifically, the model ranked first among the M models receives human
to each
of M, while the model ranked last among the M models receives human score of 1. Additionally, the model ranked among the M models assigned a human score of M k + 1. To determine the human score for each dimension, calculate the average of the human scores across all samples within that dimension. Thus, the human score for each model ranges from to M.
4 Experiments
Dimension Evaluation. For each image and instruction, we utilize official codes from various models for image editing. We calculate the I'EBench scores following the methodology described in 3.c.3.1. The E'EBench scores for original and diverse instructions are presented in Fig. 4, Tab. and Tab. 2, respectively. Our observations reveal that no single model achieves the best performance across all evaluation dimensions. Regarding low-level editing, InstructDiffusion Geng et al. [2023] demonstrates superior results. It attains the highest scores in 4 out of 7 low-level editing evaluation dimensions when using original instructions, and 3 out of 7 when using diverse instructions. For

---

Table I: I'EBench evaluation results per dimension using original instructions. Exp Min and Exp Max denote the minimum and maximum values of all samples for each evaluation dimension.
Low-level Editing
Model
Deblurring
Haze Removal
Lowlight Enhuncement
Noise Removal
Rain Removal
Shadow Removal
Snow Removal
Watermark Removal
Zhang [2023a]
44.25
InstructDiftsionn Geng et al. [2023]
42 48
56.61
28.60
67.20
37.43
55.65
88.49
InstrectPix2Pix Brooks et [2023]
48. 03
56.15
43,32
20.11
$6,64
34.19
57.59
58.12
MugicBrush Zhang al. [2024a]
59.46
37.71
20.59
60.60
63.33
MGIE Fu et al. [2024]
60.30
51.75
39.99
23.25
$6.00
36.91
34.04
55.53
InstructEdit Wang et al. [20236]
40.77
58.85
13.83
15.40
64.44
36.88
43.45
82.68
InstructAny2P%
[2023c]
34
42.97
47.28
HQ-Edit Hui et al. [2024)
35.27
39.25
41.71
22.13
38.52
33.13
38.97
29.80
Exp Min
13.79
12.66
0.09
7,38
1.05
2.18
1.34
Exp Max
91.94
92.70
89.60
77.00
96.11
89.19
89,26
96.42
High-level Editing
Model
Counting
Direction Perception
Object Removal
Replacement
Object
Background Replacement
Color Alteration
Style Alteration
Accuracy
Region
HIVE Zhang al. [2023a]
18.57
47.15 14
42.14
86.43
74.29
30.00
25.32
58.15
InstructDiffusion Geng et [2023]
15.00
44.29
65.71
42,86
60.71
53.57
21.69
66.18
InstrectPlsPPix Brooks et [2023]
13.57
37 14
25.00
44.29
65.71
49.29
23.76
61.63
MagicBrush Zhang al. [2024a]
30.71
49.29
32.14
22.78
66.34
MGIE Fu et al. [2024)
17.14
48.57
37.86
65.71
82.86
32.86
23.68
69.60
InstructEdit Wang et al. [2023b]
11.26
41.73
5,04
4.41
50.36
3.62
19.83
77.08
InstructAny2Pe Li [2023c]
20.59
52.75
HQ-FoR Edle Hul et al. [2024)
19.26
47.79
23.74
47.06
71.22
27.54
15.96
49.21
Esp Min
0,00
0.00
0.00
0.00
0.00
0.00
12.96
6.41
Exp Max
100.00
100.00
100.00
100.00
100.00
33.84
98.
Table 2: I'EBench evaluation results per dimension using diverse instructions.
Low-level Editing
Model
Deblurring
Haze Removal
Lowlight Enhuncement
Noise
Rain Removal
Shadow Removal
Snow Removal
Watermark Removal
HIVE Zhang et al. [2023u]
44.41
$4.09
42,7 28
25.51
58.59
36.69
$1.92
$7. 88
InstructDimision Geng [2023]
01
39.47
28.06
64. 68
32.54
57.30
85.14
InstructPi<PPx Brooks et [2023]
45.24
53.52
42.88
24.49
51,86
32.79
52.67
48.91
Mugielirush Zhung et al. [2024u]
45.96
55.11
33.74
23,91
55.77
36,73
54.68
59.76
MGIR Fa [2024]
57.33
$1.61
32.96
23.49
58.27
34.07
$1.02
InstructEdit Wang et [20236]
40.66
58.89
13.92
15.81
65.08
36.66
43.34
83.68
DnstractAny2PM 2P Ld et [2023c]
34.77
47 00
18.09
22.18
48.92
36.04
43.13
47.58
HQ-Edit Hui [2024]
37.60
Exp Min
6.32
3.67
0.60
0.03
7.22
1.46
3.78
2.58
Exp Max
88.17
92.69
90.34
79.29
97.05
86.27
82.24
96.39
High-level Editing
Model
Counting
Direction Perception
Ohject Removal
Object Replacement
Background Replavement
Color Alteration
Style Alterution
Region
Accuracy
HIVE Zhang et al. [2023a]
13.57
43.5 57
12,
67.86
85.00
35.00
23.08
61.97
InstrectDiftsionn Geng et [2023]
21.43
47. 76.86
22.14
47.14
64,29
48.57
19.96
65,92
InstructPixPPi Brooks et al. [2023]
18.57
47 86
7.14
47.14
65.71
43.57
23,13
61.32
MagicBrush Zhang et al. [2024a]
24.29
45.71
12.14
62.14
83.57
54.29
23.08
66.21
MGIE Fu al. [2024]
19.29
47.14
22.86
43.57
74.29
37.86
23,36
71.89
InstructEdit Wang et [30256]
11.26
3,60
4.41
$1,80
3.62
19.91
77.08
InstructAny2Fe5 2Pix Li et al. [2023c]
22.79
$1.80
43.88
48.53
68.35
12.32
25.93
$2.61
IIQ-Edit Hui al. [2024]
20.74
51.47
24.46
50.00
79.86
26.09
16.48
48.29
Esp Min
0.00
0.00
0.00
0,00
0.00
0.00
10.62
9.79
Exp Max
100.00
100.00
100.00
100.00
100.00
100.00
34.06
98.68
high-level editing, both MagicBrush Zhang et al. [2024a] and InstructAny2Pix Li et al. [2023c] perform impressively. MagicBrush achieves the highest scores in 3 evaluation dimensions using original instructions, while InstructAny2Pix achieves the highest scores in 3 dimensions using diverse instructions. In the deblurring dimensions, MGIE Fu et al. [2024] stands out significantly. It surpasses the second-place model by 11.92 when using original instructions and by 11.37 when using diverse instructions.
Human Evaluation. We ranked different models based on their E'EBench scores and computed PeBBench rank scores using the methodology described Sec. 3.3. Given that both I'EBench rank scores and human scores range from I to 8, direct comparison can be made between them. Therefore, we conducted correlation analyses and visually presented the results in Fig. 5. Significant positive correlations were observed between the EBBenh rank score and the human score across all

---

pir
jow
par
Drectinneeceetion 0.22
Sylldtheration
Resfontcourryy
p- 0.86
p-
j 0.51
per 0.38
pe 093
p- 3.56
Figure 5: Alignment between EBBehh rank scores (Y-axis) and human scores (X-axis).
300.00%
250.00%
200.00%
150.00%
50.00%
0.00%
Dabluring
Haze Removal
Lowight Entencenenn
Nelsv Remevaal
Rain Removal
Ghurlow Remeval
Brow Removal
Watemark Removal
Counting
Directon
Percaptio
Object Removal
Osjeet Repiacement
Baokground Repiacement coment
Color Ataration
Style Attertion
Region Acmunacy
HIVE
InstructD iffusion
InstructPix2Fix
MagicBrush
MGIE
InstruatEdit
InstructAny2Pix HQ-Edit
Figure 6: E'EBench change rate using original instructions and diverse instructions.
dimensions.
These findings offer strong evidence supporting the alignment between our proposed benchmark and human perception.
Insights
The editing ability across different dimensions not robust: Our observations indicate that no single model excels in all evaluation dimensions. This implies that different IIE models have varying strengths terms of their editing abilities across different dimensions. Thus, crucial to acknowledge this limitation and focus developing IIE model that demonstrates consistent and competent performance across all dimensions. Future research efforts should prioritize the creation of robust and versatile IIE model that can effectively handle wide range of editing tasks across diverse dimensions.
The editing ability of different instructions is not robust: To evaluate the robustness of editing models when provided with different instructions, we propose a metric called E'EBench change rate. This metric is defined as follows:
SS S MIN(S St
S
(1)
where St and St represent the I'EBench scores of the i-th evaluation dimension when using original and diverse instructions, respectively. The value of S indicates the I'BBench change rate for the i-th evaluation dimension. As illustrated in Fig. 6, when it comes to the object removal dimension, InstructPix2PixBrrokk et al. [2023], HIVE Zhang et al. [2023a], InstructionDiffusiinn Geng et al. [2023], and MagicBrush Zhang et al. [2024a] exhibit significant fluctuations in their performance using different instructions. On the other hand, the remaining models demonstrate relatively stable performance across different instructions. One notable distinction between these two categories of models is that the latter employs LLM Achiam et al. [2023], Touvron et al. [2023] or MLLM Liu et al. [2023b,a] to comprehend instructions, which enhances their resilience to variations in instructions. Given the unpredictable and diverse nature of user editing instructions, it is crucial to develop an editing model that can effectively handle instructions with varying levels of complexity.
The editing ability for different categories not robust: As illustrated in Fig. 7, we have observed distinct variations in the performance of different categories. Notably, the "Scenery" and "Global"

---

NIVE
HIVE
HQ-Edit
InstrussD Mlusoon
HQ-Edr
Animal
Object
Scenery
InstuctPicePss InstractAnyaPi
InstruoPaaPix
Plant
Human
Global
InstrucfEdtt
MagicBrush
InstructEdi:
MagicGrush
(a)
(b)
MGIE
MGIE
Figure 7: Comparison of radar charts for EBBechh scores different categories using (a) original instructions and (b) diverse instructions. The scores of all dimensions normalized and averaged.
categories consistently demonstrate superior performance compared to the other categories across all the IIE models we evaluated. This discrepancy can be attributed to the inherent inclination of the "Scenery" and "Global" categories towards global editing, which diminishes the necessity for precise target object localization. Given these findings, crucial to prioritize the simultaneous consideration of various editing content in future research endeavors.
Conclusions
In this paper, we present E'Bench, comprehensive benchmark specifically designed for instruction- based image editing (IIE). Our benchmark includes substantial dataset of over 2000+ images and more than 4000+ instructions, covering 16 distinct evaluation dimensions. To evaluate the effectiveness of EBBcchh we conduct experiments using 8 open-source IIE models. Additionally, we complement these experiments with meticulous human evaluations to establish the correlation between FBBcch scores and human perception. Based on the observations derived from F'EBench, we provide valuable insights and recommendations for advancing IIE models. We hope the proposed IeEBench to serve an indispensable asset, playing pivotal role in fostering the advancement of IIB models and assessing their efficacy.
Acknowledge
This work was supported by National Key R&D Program of China (N0.2023YFB45022004) the National Science Fund for Distinguished Young Scholars (No.62025603), the National Natural Science Foundation of China (No. U22B2051, No. U21B2037, No. 62072389, No. 62302411) I), the Natural Science Foundation of Fujian Province China (No.2021J06003), and China Postdoctoral Science Foundation (No. 2023M732948).
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida. Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint wrXiv:2303.08774 2023.
Codruta O Ancuti, Cosmin Ancuti, Mateu Sbert, and Radu Timofte. Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. In 2079 IEEE international conference image processing (ICIP), pages 1014-1018, IEEE, 2019.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981 17993, 2021.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 09208-82211 2022.
10

---

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Samyadeep Basu. Mehrdad Saberi, Shweta Bhardwaj. Atoosa Malemir Chegini, Daniela Massiceti, Maziar Sanjabi. Shell Xu Hu, and Soheil Feizi. Editval: Benchmarking diffusion based text-guided image editing methods. arXiv preprint arXiv:2310.02426, 2023.
James Betker, Gabriel Goh, Li Jing. Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyee Lee, Yufci Guo, et al. Improving image generation with better captions. Computer Science https://odn. openai. conépapersidall--3 pdf. 2(3):8 2023.
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji. Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333 342, 2010.
Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subhiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advonces in neural information processing systems, 33: 33:1877-1901, 2020.
Haoyu Chen, Jinjin Gu, Yihao Liu, Salma Abdel Magid, Chao Dong, Qiong Wang, Hanspeter Pfister, and Lei Zhu. Masked image training for generalizable deep image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 1692-1703, 2023a.
Wei-Ting Chen, Hao- Fang, Cheng-Lin Lin Hsich, Cheng-Che Tsai, Chen, Jian-Jiun Ding, Sy-Yen Kuo, ct al. All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4196-4205, 2021.
Xiang Chen, Hao Li, Mingqiang Li, and Jinshan Pan. Learning a sparse transformer network for effective image deraining. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5896-5905, 2023b.
Yang Chen, Yu-Kun Lai, and Yong-Jin Liu. Cartoongan: Generative adversarial networks for photo cartooniza- tion. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 9465-9474, 2018.
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang. Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint XXXv:2402.03366 2024.
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale N Fung, and Steven Hoi. Instructblip: Towards purpros- vision-langual models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: oidher-order denoising diffusion solvers. Advances in Neural Information Processing Systems, 04:30150301666 2022.
Wenkai Dong, Song Xue, Xiaoyue Duan, and Shumin Han. Prompt tuning inversion for text-driven image editing using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7430-7440, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. diternlm-xcomposee Mastering free-form text-image composition and comprehension in langua- large model. arXiv preprint arXiv:2401. 164200 2024.
Hao Fei, Shengqiong Wu. Wei Ji. Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of- thought: Step-by-step video reasoning from perception to cognition. In Forty-first International Conference on Machine Learning, 2024a.
Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat- Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. 2024b.
11

---

Hao Fei, Shenggiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhancing video- language representations with structural spatio-temporal alignment. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2024c.
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and The Gan. Guiding instruction-based image editing via multimodal large language models. International Conference Learning Representations, 2024.
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprint arXiv:2402.0523 00000, 2024.
Zigang Geng, Binxin Yang, Tiankai Hang. Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Han Hu, Dong Chen, et al. Instructdiffusion: A generalist modeling interface for vision tasks. arXiv preprint WwXiv:2309.03895 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair. Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020.
Lanqing Guo, Chong Wang, Wenhan Yang, Siyu Huang, Yufei Wang, Hanspeter Pfister, and Bihan Wen. Shadowdiffusion: When degradation prior meets diffusion model for shadow removal. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2023a.
Yun Guo, Xueyao Xiao, Yi Chang, Shumin Deng, and Luxin Yan. From sky to the ground: A large-scale benchmark and simple bascline towards real rain removal. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12097-12107, 2023b.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208. 01626. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1116880068511 2020.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint 200:22203./89 2024.
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao. Diffusion model-based image editing: A survey, arXiv preprint or 2024a.
Yuzhou Huang. Lianghin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruiman Zhang, et al. Smartedit: Exploring complex instruction- based image editing with multimodal large language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024b.
Drew A Hudson and Christopher Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang. Peng Wang. Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for instruction-based image editing. arXiv preprint arXiv:2404.0990 2024.
Jiayi Ji, Yiwei Ma, Xiaoshuai Sun, Yiyi Zhou, Yongjian Wu, and Rongrong Ji. Knowing what learn: a metric- oriented focal mechanism for image captioning. IEEE Transactions on Image Processing, 31:4321 1-335,, 2022.
Tero Karras, Samuli Laine, and Timo Aila, A style- generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 33:12104 12114, 2020a.
12

---

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVT conference on computer vision and puttern recognition, pages 8110-8119, 2020b.
Bahjat Kawar, Michael Flad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593- 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang. Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong Reflash dropout in image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2022.
Jari Korhonen and Junyong You. Peak agnal-to-noise ratio revisited: Is simple bcautiful? In 2012 Fourth International Workshop Quality of Multimedia Experience, pages 37-38. IEEE, 2012.
Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: A single image denoising diffusion model. In International Conference on Machine Learning (ICML), pages 77920-17930 PMLR, 2023.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-hench: Benchmarking multimodal IIms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, al.
Mybench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.1000 2023b.
Shufan Li, Harkanwar Singh, and Aditya Grover. Instructany2pix: Flexible visual editing via multimodal instruction following. arXiv preprint arXiv:3112.06338 2023c.
Yifan Li. Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large rision-languasl models. arXiv preprint or Xiv::2305 10355, 2023d.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision- EooonEECV// 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pages 740-755. Springer. 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning, 2023b.
Yang Liu, Zhen Zhu, and Xiang Bai Wdnet: Watermark
network for visible watermark removal. In 2021 IEEE/CVF Winter Conference Applications of Computer Vision (WACV). IEEE, 2021a.
Ye Liu, Lei Zhu, Shunda Pei, Huazhu Fu, Jing Qin, Qing Zhang, Liang Wan, and Wei Feng. From synthetic to real: Image dehazing collaborating with unlabeled real data. In Proceedings of the 29th ACM international conference multimedia, pages 58, 2021b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoming via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507 25211 2022.
Yiwei Ma. Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to-end multi-grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 638-647, 2022.
Yiwei Ma, Jiayi Ji, Xiaoshuai Sun, Yiyi Zhou, and Rongrong Ji. Towards local visual modeling for image captioning. Pattern Recognition, 138: 109420, 2023.
Yiwei Ma. Zhibin Wang. Xiaoshuai Sun, Weihuang Lin, Qiang Zhou, Jiayi Ji, and Rongrong Ji. Inf-llava: Dual- perspective perception for high-resolution multimodal large language model. arXiv preprint 2407 16/98, 2024.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang. and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2017.
13

---

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedingy of the IEEE/cvf conference computer vision and patter recognition, pages 3195-3204, 2019.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2. pages 416 423, July 2001.
Scungjun Nah, Tac Hyun Kim, and Kyoung Mu Lec. Deep multi-scale convolutional neural network for dynamic scene deblurring, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In PMLR, 2021.
International conference machine learning, pages
Lianggiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson WH Lau. Deshadownet: A multi- context embedding deep network for shadow removal. In Proceedings of the IEEE conference on computer vision and patter recognition, pages 4067-4075, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML), pages 8748-8766 PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iyya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXi::220.006125, 1())3, 2022.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy I Lillicrap, cean-baptiste Alayrac, Radu Soricut, Angeliki I Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
Chitwan Saharia, William Chan. Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479- 36494, 2022.
Yash Sanghvi, Zhiyuan Mao, and Stanley H Chan. Structured kemel estimation for photon- deconvolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9863-9872, 2023.
Eric Saund, David Fleet, Daniel Larner, and James Mahoney. Perceptually-supporre image editing of text and graphics. In Proceedings of the 16th annual ACM symposium on User interface software and technology. pages 183-192, 2003.
Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5572-5581 5581, 2019.
Shelly Sheynin, Adam Polyak. Uriel Singer. Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. arXiv preprint arXiv:2311.1008, 2023.
Jascha Sohk-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning (ICML), pages 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2xX 2023.
14

---

Dani Valevski. Matan Kalman, Eyal Molad, Eyal Segalis, Yossi Matias, and Yaniv Leviathan. Unitune: Text- driven image editing by fine tuning diffusion model single image. ACM Transactions on Graphics (TOG), 120 2023.
Hang Wang, Xuanhong Chen, Bingbing Ni, Yutian Liu, and Jinfan Liu. Omni aggregation networks for lightweight image super-resolution. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 22378-22387, 2023a.
Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka. Instructedit: Improving automatic masks for diffusion-based image editing with user instructions. arXiv preprint arXiv:2305. 18047, 2023b.
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and edithench; Advancing and evaluating text-guided image inpainting. In Proceedings of the IEEFJCVE conference on computer vision and pattern recognition, pages 18359-18369, 2023c.
Yabing Wang, Jianfeng Dong, Tianxiang Liang, Minsong Zhang, Rui Cai, and Xun Wang. Cross- lingual cross-modal retrieval with noise-robust learning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 422-433, 2022.
Yabing Wang. Fan Wang. Jianfeng Dong, and Hao Luo. C12em: Improving cross -ingual cross cross-modal retrieval via cross-lingual knowledge transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5651-5659, 2024a.
Yabing Wang. Shuhui Wang. Hao Luo, Jianfeng Dong, Fan Wang, Meng Han, Xun Wang, and Meng Wang. Dual-view curricular optimal transport for cross-lingual cross-modal retrieval. IEEE Transactions on Image Processing, 33:1522-1555 1533, 2024b.
Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-61) 2004.
Chen Wei, Wenjing Wang, Wenhan Yang, and Jaaynng Liu. Deep retinex decomposition for low-light enhance- ment. arXiv preprint arXiv: 1808.04566 2018
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML), pages 681 681-688, 2011.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.0467 2023a.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang. Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, al. Q-bench: benchmark for gencral-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.1 141881 2023b.
Rui-Qi Wu, theng-Peng Duan, Chun-Le Guo, Zhi Chai, and Chongyi Li. Ridep: Revitalizing real image dehazing via high-quality codebook priors. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 22282-2229 2023c.
Sihan Xu, Ziqiao Ma, Yidong Huang, Honglak Lec, and Joyce Chai. Cyclenet: Rethinking cycle consistency in text-guided diffusion for image manipulation. Advances in Neural Information Processing Systems, 36, 2024.
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
WeiHao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Li- juan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint 200.::20080024900 2023.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pages 7354-7363. PMLR, 2019.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: manually annotated datasct for instruction- guided image editing. Advances in Neural Information Processing Systems, 36, 2024a.
Richard Zhang. Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595 2018.
15

---

Shu Zhang, Xinyi Yang. Yihao Feng. Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. arXiv preprint arXiv:2303, 09.18888, 2023a.
Zhixing Zhang, Ligong Han, Arna Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. Proceedings of the IEEE/CVE Conference on Computer Vision Puttern Recognition, pages 2037, 2023b.
Zhongping Zhang, Jian Zheng. Zhiyuan Fang, and Bryan A Plummer. Text-to-image editing image information removal. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5232-5241, 2024b.
Dongsheng Zhu, Xunzhu Tang, Weidong Han, Jinghui Lu, Yukun Zhao, Guoliang Xing, Junfeng Wang, and Dawci Yin. Vislinginstruct: Elevating zero-shot lcarning in multi-modal language models with autonomous instruction optimization. arXiv preprint MXix:2402.00396 2024.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The last paragraph of the abstract and introduction explains the contribution and scope of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification:
The limitation section is included in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
All theoretical results have been fully proven.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper provides sufficient details to reproduce the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code and dataset are included in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /Cod/Soodemiiiceeeee SooodeeumisssooPPoi
/ //nips... /tippscc/pputt wttp:cccpppppppcccccceo/ococpcca
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification:
All replicates of IIE models are sourced from official settings and code.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, the appendix, or supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Our goal to propose benchmark, so there is no need for Experiment Statistical Signature.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: We did not train the IIE model, and all checkpoints are sourced from the official code, so there is no need to report them.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpe://eeurssscceooeeoo https://earrppp.ccpppll////
Answer: [Yes]
Justification: Research conducted in the paper conform, every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
21

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have made appropriate citations in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcome com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
The proposed dataset are included in Supplementary materials.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
22

---

Answer: [Yes]
Justification: A screenshot of the human evaluation questionnaire can be found in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our experiment meets the requirements.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "x2nSSQCkM0", "reviewer": "Reviewer_riYa", "summary": "This paper proposes I2EBench, a comprehensive benchmark designed to automatically evaluate the quality of edited images produced by IIE models from multiple dimensions. I2EBench comprises 16 evaluation dimensions, covering both high-level and low-level aspects. Additionally, through user studies, the authors assess the alignment between the proposed benchmark and human perception. The I2EBench dataset consists of over 2000 images for editing, along with corresponding original images and diverse instructions.", "questions": "1. Does I2EBench consider the aesthetic quality of image edits? For example, in Object Replacement (line 164), how does the benchmark evaluate whether the edited object is appropriately and naturally integrated into the image, rather than simply copied and pasted?", "limitations": "The authors adequately addressed the limitations in the manuscript.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed I2EBench represents a significant advancement over previous works, providing a large and comprehensive benchmark for instruction-based image editing. This contribution will greatly benefit the research community.\n2. When establishing I2EBench, the authors have thoughtfully included often overlooked low-level edits, such as rain removal, and have implemented thorough evaluations.\n3. The paper is clearly written and easy to follow.", "weaknesses": "1. The technical novelty of the evaluation method is insufficient. A significant part of I2EBench's evaluation of high-level edits relies on GPT-4V, such as Direction Perception (line 158) and Object Removal (line 164). Therefore, the authors' contribution appears more like a new prompt engineering method.\n\n2. The insights provided in Section 5 are not informative for the research community. The authors state that \"the editing ability across different dimensions is not robust,\" and Figure 7 shows that current IIE methods perform better on high-level editing tasks (e.g., Object Removal) but struggle with low-level tasks (e.g., Shadow Removal). However, most existing editing datasets are high-level, making it difficult for IIE methods to learn low-level tasks during training. Therefore, the insights in Section 5 seems not constructive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your positive feedback on our paper and for recognizing its strengths. We appreciate your agreement that I2EBench represents a significant advancement over previous works, greatly benefiting the research community. Additionally, we are thankful for your observation that we have thoughtfully included often overlooked low-level edits. We are also grateful for your praise regarding the clarity and readability of our paper.\n\nNow, let us address the specific weaknesses and questions you have raised and provide further clarification:\n\n> ***Q1:** The technical novelty of the evaluation method is insufficient.*\n> \n\n**A1:** \n\nThank you very much for your insightful feedback. In fact, GPT-assisted evaluation methodologies are widely recognized and utilized in numerous influential published works [1,2,3], demonstrating considerable scientific validity. While it is true that we leverage GPT-4V for certain aspects of the evaluation, it is crucial to understand that our approach represents more than mere prompt engineering. To elucidate further, the effective utilization of GPT-4V in our benchmark necessitated several significant contributions:\n\n- **Image Selection:** For each evaluation dimension, we meticulously select and filter suitable images. For example, for the evaluation dimensions of \"Object Removal\" and \"Object Replacement\", it is necessary to ensure that there is at least one object in the image instead of a simple landscape photo. This rigorous curation process ensures the relevance and accuracy of the images used, thereby enhancing the integrity of our evaluation.\n- **Instruction Annotation:** Each image is annotated with a specific editing instruction that aligns precisely with the corresponding evaluation dimension. This careful annotation is essential for ensuring that the assessments are accurate and meaningful.\n- **Question-Answer Pairing for High-Level Edits:** For dimensions involving high-level edits, we annotate each image-instruction pair with corresponding question-answer pairs. The accuracy of the edits is then evaluated by comparing GPT-4V's responses to these annotated answers. This process allows us to rigorously assess the correctness and effectiveness of the edits.\n\nThus, I2EBench should be viewed not merely as a prompt engineering method but as a well-designed, thoroughly annotated benchmark. It leverages GPT-4V to reduce evaluation costs and enable the automation of the evaluation process, which is in line with current practices in the field.\n\n*[1] Q-bench: A benchmark for general-purpose foundation models on low-level vision. ICLR. 2023.*\n\n*[2] Evalcrafter: Benchmarking and evaluating large video generation models. CVPR. 2024.*\n\n*[3] Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS. 2023.*\n\n> ***Q2:** The insights provided in Section 5 are not informative for the research community.*\n> \n\n**A2:** \n\nThank you for your constructive feedback.\n\nWe agree that the current emphasis on high-level tasks in existing IIE datasets may limit the capacity of IIE methods to effectively learn and execute low-level editing tasks. This limitation actually reinforces the validity and reliability of our analysis. Our proposed I2EBench aims to address this gap by offering a comprehensive evaluation framework that includes both high-level and low-level editing tasks.\n\nTherefore, far from being non-informative, we believe our proposed insights serve as a crucial signal to the research community, highlighting the clear need for balanced dataset development and methodological advancements that can better handle low-level tasks. These insights provide a clear direction for future research and innovation, emphasizing areas that require further exploration and improvement. Your valuable comments have enabled us to more effectively convey the significance and implications of our findings.\n\nIf you have any questions or need further clarification, please do not hesitate to let us know, and we will do our utmost to address them.\n\n> ***Q3:** Does I2EBench consider the aesthetic quality of image edits?*\n> \n\n**A3:** \n\nThank you for your insightful comments. Aesthetic quality is indeed an important criterion in image editing. In response to your suggestion, we have integrated the Aesthetic Predictor’s Score (AP) to evaluate the aesthetic quality of edited images, similar to the approach used by InstructDiffusion [1]. Specifically, the AP score assesses the aesthetic quality of the generated images, employing a methodology akin to that used by LAION-5B [2], which utilizes the CLIP+MLP Aesthetic Score Predictor. A higher AP score indicates a better perceptual quality.\n\nWe calculated the AP score for the edited images generated by different methods and averaged the scores for each evaluation dimension. This allows us to derive the AP scores for each method across various evaluation dimensions. As illustrated in **Tab. I and Tab. II of the rebuttal pdf**, our findings are twofold:\n\n- The difference in the Aesthetic Predictor’s Score between images edited using original instructions and those edited with diverse instructions is relatively small.\n- The variations in the Aesthetic Predictor’s Score across different dimensions are relatively large.\n\nThese insights will enable us to refine our evaluation methodology and ensure a more comprehensive assessment of the aesthetic quality of image edits. Following your excellent suggestion, we will include the above experimental results and discussion in the revised manuscript.\n\n*[1] Instructdiffusion: A generalist modeling interface for vision tasks. CVPR. 2024.*\n\n*[2] Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS. 2022.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ood0ELpm0d", "reviewer": "Reviewer_7DEf", "summary": "The paper addresses the challenge of evaluating models in the field of Instruction-based Image Editing (IIE) by proposing a comprehensive benchmark called I2EBench. It features: 1) Comprehensive Evaluation: Covers 16 evaluation dimensions for a thorough assessment of IIE models. 2) Human Perception Alignment: Includes extensive user studies to ensure relevance to human preferences. 3) Research Insights: Provides analysis of strengths and weaknesses of existing IIE models to guide future development. I2EBench will be open-sourced, including all instructions, images, annotations, and a script for evaluating new models.", "questions": "1 Why did the authors choose to sample high-level editing images from the COCO dataset, while most low-level editing images are sampled from existing low-level datasets?\n\nOthers please ref to the weaknesses.", "limitations": "yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1 The supplementary materials provided by the authors offer an in-depth explanation of the evaluation process, which is very beneficial for readers seeking to understand the evaluation details of I2EBench thoroughly.\n\n2 The benchmark proposed in the paper covers a wide range of evaluation dimensions, providing a holistic and multi-faceted assessment of IIE model capabilities.\n\n3 The authors commit to open-sourcing I2EBench, including all relevant resources. This openness will facilitate fair comparisons and knowledge sharing within the community.\n\n4 By systematically evaluating the models, the paper provides valuable research insights that can guide future model architecture design and data selection strategies.\n\n5 The inclusion of user studies in the evaluation process adds depth, ensuring that the evaluation results accurately reflect the real-world experiences of end-users.\n\n6 The benchmark encompasses multiple types of image editing tasks, including both high-level and low-level editing, which makes it versatile and comprehensive.\n\n7 A large number of images and instructions are provided, forming a solid foundation for comprehensive testing of the models.", "weaknesses": "1 If the research code or data is not made publicly available, it may limit the usability of the benchmark for other researchers.\n\n2 The paper only provides radar charts for the category experiments. Including the corresponding quantitative data would make the comparisons more precise and understandable.\n\n3 I observed a significant performance gap in the Object Removal dimension between using the original instruction and diverse instruction. The authors could use methods like CLIP similarity or Jaccard similarity to measure the similarity between the original and diverse instructions. This could help determine whether the performance variance is due to significant changes in the instructions or due to the sensitivity of some models to specific vocabulary changes in the Object Removal instructions.\n\n4 The font size in Figure 3 (c) is too small, which might hinder readers from clearly viewing the information presented in the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our heartfelt gratitude for your thoughtful feedback on our paper. We sincerely appreciate your recognition of its strengths, including the comprehensive explanation of the evaluation process, the wide array of evaluation dimensions, the provision of evaluation code, and the valuable research insights we have provided. Additionally, we are thankful for your acknowledgment that I2EBench is a holistic benchmark encompassing both low-level and high-level editing tasks.\n\nNow, let us address the specific weaknesses and questions you have highlighted, and provide further clarification:\n\n> ***Q1:** If the research code or data is not made publicly available, it may limit the usability of the benchmark for other researchers.*\n> \n\n**A1:** \n\nThank you for raising this important concern.\n\nWe fully appreciate that the accessibility of research code and data is crucial for the usability and reproducibility of our benchmark by other researchers. In response to this, we have included the code in the supplementary materials accompanying our submission. Additionally, we are committed to fully open-sourcing both the code and data.\n\n> ***Q2:** The paper only provides radar charts for the category experiments. Including the corresponding quantitative data would make the comparisons more precise and understandable.*\n> \n\n**A2:** \n\nThank you for your insightful suggestion.\n\nTo provide a more precise and comprehensible comparison of our benchmark's performance across different categories, we have included the corresponding quantitative data. **Tab. III and Tab. IV of the rebuttal pdf** present the quantitative results for different model categories, supplementing the radar charts displayed in Figure 7. These detailed metrics will be incorporated into the supplementary materials of the final version of our paper to enhance clarity and further substantiate our findings.\n\n> ***Q3:** I observed a significant performance gap in the Object Removal dimension between using the original instruction and diverse instruction. The authors could use methods like CLIP similarity or Jaccard similarity to measure the similarity between the original and diverse instructions. This could help determine whether the performance variance is due to significant changes in the instructions or due to the sensitivity of some models to specific vocabulary changes in the Object Removal instructions.*\n> \n\n**A3:** \n\nThank you for your valuable suggestion.\n\nFollowing your recommendation, we have employed multiple metrics to measure the similarity between the original and diverse instructions, including CLIP cosine similarity, Jaccard similarity, TF-IDF, Word2Vec, and FastText. **Tab. V of the rebuttal pdf** presents the similarity scores, illustrating that there are no significant abnormalities across these metrics when comparing the original and diverse instructions in the Object Removal dimension. Thus, the observed performance gap in the Object Removal dimension is not due to textual discrepancies. Instead, it indicates that these models exhibit a lack of robustness in interpreting the instructions for this particular task.\n\n> ***Q4:** The font size in Figure 3 (c) is too small, which might hinder readers from clearly viewing the information presented in the figure.*\n> \n\n**A4:** \n\nThank you for your valuable feedback.\n\nWe understand that the readability of figures is crucial for conveying information effectively. In response to your observation, we will revise Figure 3(c) to increase the font size, ensuring that all text is clearly legible.\n\n> **Q5:** Why did the authors choose to sample high-level editing images from the COCO dataset, while most low-level editing images are sampled from existing low-level datasets?\n> \n\n**A5:** \n\nThank you for your insightful question.\n\nFor low-level editing tasks, we sourced images from existing low-level datasets because these datasets provide ground truth (GT) images. The availability of GT images allows for a more precise evaluation by directly calculating metrics such as SSIM (Structural Similarity Index) between the GT images and the edited images, ensuring accuracy in assessment.\n\nIn contrast, high-level editing tasks often lack clearly defined ground truth images, making it challenging to perform precise evaluations. Therefore, we chose to sample high-level editing images from the COCO dataset, a widely recognized and accepted dataset in the research community. The use of COCO ensures a broad and diverse range of images, facilitating a more representative assessment of high-level editing capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m3qvWpzjKU", "reviewer": "Reviewer_A2Kh", "summary": "This paper proposes I2EBench, which is a new benchmark for evaluating Instruction-based Image Editing (IIE) models. It offers a large dataset with over 2,000 images and 4,000 instructions across 16 detailed evaluation dimensions. The benchmark is designed to assess image editing quality automatically and aligns with human perception through extensive user studies. I2EBench aims to provide insights for improving IIE models and will be open-sourced for community use.", "questions": "I would like to inquire if the I2EBench tool or framework will be open-sourced in the future. This information is crucial for understanding its potential accessibility and contributions to the community.", "limitations": "Yes. The limitations of the paper described by the author in the appendix.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I commend the authors for their insightful paper, particularly the proposed benchmark, which is remarkably comprehensive and has the potential to significantly advance the field of instruction-based image editing (IIE).\n1.The benchmark addresses a wide array of evaluation dimensions, offering a holistic and multifaceted assessment of IIE model capabilities. This comprehensive approach ensures that the strengths and weaknesses of various models are thoroughly examined from multiple perspectives.\n2.It strongly emphasizes aligning with human perception, ensuring the benchmark's relevance to human preference.\n3.The large collection of images and instructions forms a solid foundation for comprehensive testing. This large and diverse dataset provides a robust foundation for thorough and rigorous testing, enabling models to be evaluated across a broad spectrum of scenarios.", "weaknesses": "Although the paper proposes a amazing benchmark, I think the following points can further improve the paper:\n1.The results of multiple models on the benchmark were presented in the paper; however, the specifics of their evaluation were not described in sufficient detail. For instance, information regarding the hyperparameters used for the evaluation process and the types of GPUs employed for model testing would provide greater clarity and reproducibility.\n2.The rationale behind the use of GPT-4V for supplementary evaluation requires further elucidation. Additionally, whether alternative models could serve as suitable substitutes for GPT-4V?\n3.In the section on Instruction, the contributions of the paper are not sufficiently highlighted. A more thorough summarization of the paper's contributions would significantly aid in conveying its impact to the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our heartfelt gratitude for your thoughtful and encouraging feedback on our paper. We are deeply appreciative of your commendation, recognizing our work as insightful and valuable. Thank you for acknowledging that our benchmark encompasses a wide range of evaluation dimensions and places a strong emphasis on alignment with human perception. Additionally, we are grateful for your praise regarding the extensive collection of images and instructions, which indeed forms a robust foundation for comprehensive testing.\n\nNow, let us address the specific weaknesses and questions you have highlighted and provide further clarification:\n\n> ***Q1:** The results of multiple models on the benchmark were presented in the paper; however, the specifics of their evaluation were not described in sufficient detail. For instance, information regarding the hyperparameters used for the evaluation process and the types of GPUs employed for model testing would provide greater clarity and reproducibility.*\n> \n\n**A1:** \n\nThank you for your valuable feedback. We utilized the hyperparameter settings and official weights as specified in the original papers for each model. All editing results were generated using A800 GPUs.\n\nWe will include these implementation details in the revised version of our paper to enhance transparency and facilitate the reproducibility of our results.\n\n> ***Q2:** The rationale behind the use of GPT-4V for supplementary evaluation requires further elucidation. Additionally, whether alternative models could serve as suitable substitutes for GPT-4V?*\n> \n\n**A2:** \n\nThank you for your insightful question.\n\nOur decision to use GPT-4V in our supplementary evaluation was based on its status as the most advanced model available at the time of our study. At that point, GPT-4o had not yet been proposed. We chose GPT-4V for its sophisticated capabilities, including its nuanced understanding of language and proficiency in handling complex tasks. As technology continues to evolve, it is indeed possible that other multimodal large language models may emerge, offering even more accurate assessments in the future.\n\n> ***Q3:** In the section on Instruction, the contributions of the paper are not sufficiently highlighted. A more thorough summarization of the paper's contributions would significantly aid in conveying its impact to the readers.*\n> \n\n**A3:** \n\nThank you for your valuable feedback. In light of your comments, we have outlined our key contributions as follows:\n\n- We have proposed a comprehensive benchmark encompassing 16 evaluation dimensions specifically designed to assess instruction-based image editing (IIE) tasks.\n- We have conducted extensive experiments on eight popular IIE models, accompanied by a thorough analysis of their performance.\n- We have implemented Alignment Verification, demonstrating that our benchmark scores are aligned with human perception.\n\nWe will incorporate this summarized list of contributions into the revised manuscript to ensure that the readers can clearly understand the impact and significance of our work.\n\n> ***Q4:** I would like to inquire if the I2EBench tool or framework will be open-sourced in the future. This information is crucial for understanding its potential accessibility and contributions to the community.*\n> \n\n**A4:** \n\nThank you for your suggestion. The evaluation scripts for I2EBench have already been included in the supplementary materials. We are committed to open-sourcing all I2EBench codes and datasets to foster progress in this field. This initiative will ensure the accessibility and usability of our tool for the broader research community, promoting further advancements and collaborative developments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pBJvGiqFyW", "reviewer": "Reviewer_HHca", "summary": "This paper presents a comprehensive benchmark for instructional image editing. The benchmark contains a high-quality dataset with over 2000 images and 4000 instructions. In addition, the benchmark presents a new evaluation pipeline that leverages GPT to act as the judge to validate the performance of different instructional image editing methods. Within the evaluation, there are two major factor groups considering both high-level and low-level editing.", "questions": "I am confused about the definition of low-level image editing, which is actually restoration and enhancement. \n\nFirst, low-level restoration tasks usually involve fine-grained operations with rich details. Can GPT really see the slight difference when two similar restored results only vary slightly in terms of visual observation or PSNR evaluation? \n\nSecond, many editing methods are not designed for low-level vision tasks. It is not appropriate to use them to perform these tasks. The authors should check more methods specially designed for these tasks.\n\nI hope the authors can perform additional experiments to address the above concerns. For example, for low-light image enhancement, using two leading methods (such as [2] and [3]) to enhance one image with similar outputs visually (or similar PSNR) and then using GPT for evaluation.\n\n[2] Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement (https://arxiv.org/abs/2303.06705)\n\n[3] Low-Light Image Enhancement with Wavelet-based Diffusion Models (https://arxiv.org/abs/2306.00306)", "limitations": "The paper should be submitted to the NeurIPS Dataset and Benchmark track.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Due to the lack of high-quality benchmarks in image editing, this work fills the gap and is of great significance to the community. The evaluation pipeline is reasonable and clear to follow. The idea of using multimodal large language models to evaluate image editing results is interesting. The overall presentation is clear.", "weaknesses": "1. The provided Google Drive link cannot be opened, making the benchmark images inaccessible for review.\n\n2. Image editing evaluation remains a challenge due to few benchmarks. However, the authors ignore comparing their work with a related work [1] due to similar technical pipelines used with that work. Both this and that work first perform human collection, then automated evaluation using GPT, then human evaluation, and then alignment evaluation. The authors should explain the missing comparison.\n\n3. In the high-level editing, the authors did not discuss the evaluation dimension of action change or shape and size change, which are also quite essential editing types.\n\n[1] Diffusion Model-Based Image Editing: A Survey. https://arxiv.org/abs/2402.17525", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our heartfelt gratitude for your thoughtful and encouraging feedback on our paper. We are deeply appreciative of your commendation, recognizing our work as a significant contribution that fills a notable gap in high-quality benchmarks for image editing. Your praise regarding the reasonableness and clarity of the evaluation pipeline, as well as the innovative use of multimodal large language models to assess image editing results, is highly encouraging. Additionally, we are grateful for your positive remarks on the overall clarity of our presentation.\n\nNow, let us address the specific weaknesses and questions you have highlighted and provide further clarification:\n\n> ***Q1:** The provided Google Drive link cannot be opened, making the benchmark images inaccessible for review.*\n> \n\n**A1:** \n\nWe sincerely apologize for the inconvenience caused by the inaccessibility of the Google Drive link. Upon investigation, we discovered that our anonymous Google account was mistakenly suspended by Google's system on June 21, 2024, resulting in the data on Google Drive becoming inaccessible. We have since appealed this suspension, and the Google Drive link is now accessible once again. If you have any further questions or encounter any other issues, please do not hesitate to contact us.\n\n> Q2: Image editing evaluation remains a challenge due to few benchmarks. However, the authors ignore comparing their work with a related work [1] due to similar technical pipelines used with that work.\n> \n> \n> *[1] Diffusion Model-Based Image Editing: A Survey.*\n> \n\n**A2:** \n\nThank you for pointing out the relevant and intriguing paper. We apologize for not referencing and comparing our work with this study prior to submission. We will incorporate a thorough discussion and comparison with this work in our revised manuscript. Specifically, the comparison between our benchmark and the one presented in [1] can be outlined as follows:\n\n1. **Evaluation Methodology:**\n    - **Our Approach:** For each evaluation dimension, we meticulously designed questions to compare GPT’s responses with annotated answers to assess the accuracy of the edits.\n    - **[1]’s Approach:** The related work uses GPT to score the editing results.\n2. **Number of Dimensions:**\n    - **Our Benchmark:** We propose a comprehensive benchmark with 16 high-level and low-level dimensions to thoroughly evaluate different types of edits.\n    - **[1]’s Benchmark:** The related work covers 8 dimensions.\n3. **Number of Images per Dimension:**\n    - **Our Benchmark:** I2EBench includes approximately 120 images for each dimension, providing robust validation for each type of edit.\n    - **[1]’s Benchmark:** The related work includes 50 images per dimension.\n\nWe will ensure these detailed discussions are included in the final version of our manuscript. Your suggestion will undoubtedly contribute to a more comprehensive and complete presentation of our work.\n\n> ***Q3:** In the high-level editing, the authors did not discuss the evaluation dimension of action change or shape and size change, which are also quite essential editing types.*\n> \n\n**A3:** \n\nThank you for your valuable feedback. You are absolutely correct that the evaluation dimensions of action change and shape and size change are crucial for a comprehensive assessment of high-level editing. We appreciate your insightful recommendation.\n\nIn light of your suggestion and the mentioned related works[1], we will actively incorporate these evaluation dimensions into our benchmark in future iterations. This will ensure that all essential aspects of high-level editing are thoroughly evaluated, ultimately contributing to more accurate and useful benchmarking for the research community.\n\n> ***Q4:** First, low-level restoration tasks usually involve fine-grained operations with rich details. Can GPT really see the slight difference when two similar restored results only vary slightly in terms of visual observation or PSNR evaluation? …  For example, for low-light image enhancement, using two leading methodsto enhance one image with similar outputs visually (or similar PSNR) and then using GPT for evaluation.*\n> \n\n**A4:** \n\nThank you for your insightful question. It appears there may have been some misunderstanding regarding our evaluation methodology for low-level restoration tasks. As indicated in lines 186-187 of our manuscript, we employed the Structural Similarity Index (SSIM) between the ground truth image and the edited image to measure the quality of low-level edits. Consequently, we did not rely on GPT-4V for this aspect of the evaluation. Therefore, whether GPT can discern slight differences when two similar restored results vary minimally in visual observation or PSNR evaluation does not impact the accuracy of our low-level evaluation.\n\nTo prevent further misunderstandings, we will clarify our evaluation methodology as detailed on lines 186-187 in the revised manuscript. \n\n> ***Q5:** Second, many editing methods are not designed for low-level vision tasks. It is not appropriate to use them to perform these tasks.*\n> \n\n**A5:** \n\nThank you for your insightful suggestion.\n\nFor various independent low-level tasks, there are already well-established datasets in existence. Models specifically designed for these tasks can be appropriately evaluated using these datasets.\n\nHowever, the advent of large models capable of handling multiple tasks has become a significant trend in the field. Our primary objective is to encourage research on versatile models that can support both high-quality high-level and low-level image editing. Therefore, in designing our benchmark, we focused on assessing the capabilities of general-purpose editing models across a range of evaluation dimensions, encompassing both high-level and low-level editing capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### Response To All Reviewers and Table data

We would like to extend our heartfelt gratitude to the reviewers for their valuable feedback and positive comments on our paper. Their insightful reviews have significantly enhanced the clarity and overall quality of our work.

We thank Reviewer riYa  $\color{red}{\mathbf{(5: Borderline Accept)}}$ for acknowledging the strengths of our paper. They highlighted the clear and easy-to-follow nature of our writing and recognized that I2EBench represents a significant advancement over previous works, benefiting the research community. They also appreciated our thoughtful inclusion of often overlooked low-level edits.

Reviewer 7DEf  $\color{red}{\mathbf{( 6: Weak Accept)}}$ praised our comprehensive explanation of the evaluation process and the wide range of evaluation dimensions covered by the benchmark. They also commended the valuable research insights provided, which can guide future model architecture design and data selection strategies. Additionally, they noted that the benchmark encompasses multiple types of image editing tasks.

Reviewer A2Kh $\color{red}{\mathbf{(7: Accept)}}$  recognized the remarkable comprehensiveness of I2EBench and its potential to significantly advance the field of instruction-based image editing. They appreciated the benchmark's wide array of evaluation dimensions and its strong emphasis on aligning with human perception.

Reviewer HHca  $\color{red}{\mathbf{(4: Borderline Reject)}}$ highlighted the novelty and efficacy of our proposed technical components. They acknowledged the importance of our work in addressing the lack of high-quality benchmarks in image editing. Furthermore, they found the evaluation pipeline reasonable and clear to follow, praised the innovative use of multimodal large language models for evaluating image editing results, and appreciated the overall clarity of our presentation.

We sincerely thank the reviewers for recognizing these strengths and for their positive feedback on the clarity, novelty, and effectiveness of our proposed methods. Their comments have further motivated us to address the concerns and improve upon the weaknesses pointed out in their reviews. We are dedicated to thoroughly addressing their concerns and providing a detailed response in our rebuttal. ***The tables involved in the rebuttal are all located in the PDF submitted during the rebuttal phase.***

Meta-review: This paper was reviewed by four experts in the field.  The paper received mixed reviews: 5, 6, 8, 8. All reviewers find that the authors are addressing an important problem and the proposed evaluation datasets and metrics are important for the image editing community, and the analysis is thorough. AC also agrees that this is well-presented and important work and thus would suggest acceptance.

Still, this work remains some issues. Particularly, the usage of GPTv4 for supplementary evaluation requires further justification and the authors should also mention that as a potential limitation in the conclusion. Also, it shows a limited performance on low-level tasks, compared with high-level ones. Given that, we think there are still a lot of room for improvement and thus only recommended for poster presentation.

We strongly recommended the authors to carefully read all reviewers’ final feedback, and revise the manuscript as suggested in the final camera-ready version. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)