SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MAGICBRUSH: A Manually Annotated Dataset for Instruction-Guided Image Editing
Kai Zhang $^{1*}$  Lingbo Mo $^{1*}$  Wenhu Chen $^{2}$  Huan Sun $^{1}$  Yu Su $^{1}$
<sup>1</sup>The Ohio State University <sup>2</sup>University of Waterloo
{zhang.13253, mo.169, su.809}@osu.edu
https://osu-nlp-group.github.io/MagicBrush
"Let the cat
"Have him a cowboy hat"
"Let it be angry
"Change the
shirt to plaid?
have blue eyes'
"Add a barn in
and hiss
"Add a bale of
"Add a stream
the background".
(a) Source Image
Figure 1: MAGICBRUSH provides 10K manually annotated real image editing triplets (source image, instruction, target image), supporting both single-turn and multi-turn instruction-guided editing.
(b) Target Image in Turn
Multi-turn Editing
by the sheep"
(c) Target Image in Turn 2
(d) Target Image in Turn 3
"Remove the
ut a smiley face.
(a) Source Image
(b) Target Image
Single-turn Editing
Text-guided image editing is widely needed in daily life, ranging from personal use to professional applications such as Photoshop. However, existing methods are either zero-shot or trained on an automatically synthesized dataset, which contains a high volume of noise. Thus, they still require lots of manual tuning to produce desirable outcomes in practice. To address this issue, we introduce MAGICBRUSH (https://osu-nlp-group.github.io/MagicBrush/), the first large-scale, manually annotated dataset for instruction-guided real image editing that covers diverse scenarios: single-turn, multi-turn, mask-provided, and mask-free editing. MAGICBRUSH comprises over 10K manually annotated triplets (source image, instruction, target image), which supports training large-scale text-guided image editing models. We fine-tune InstructPix2Pix on MAGICBRUSH
and show that the new model can produce much better images according to human evaluation. We further conduct extensive experiments to evaluate current image editing baselines from multiple dimensions including quantitative, qualitative, and human evaluations. The results reveal the challenging nature of our dataset and the gap between current baselines and real-world editing needs.
Applying non-trivial semantic edits to real photos has long been an interesting task in image processing [27]. With the ever-increasing demand for visual content, image editing has become even more essential for enhancing and manipulating images in various fields including photography, advertising, and social media. Natural language, as our innate and flexible interface, serves as an easy way to guide the image editing process. As a result, text-guided image editing [21, 3, 8, 16, 14] has recently gained more popularity compared to other mask-based image editing techniques [20, 35, 23].
Many text-guided image editing methods have been proposed recently and achieved impressive results. These methods can be roughly divided into two categories: (1) zero-shot editing [2, 1, 24], these pipeline methods require massive amount of manual tuning of its hyperparameters to produce reasonable results. (2) end-to-end editing trained on synthetic datasets [4, 37, 7]. However, such silver training data may not only contain annotation errors but also not well capture the need and diversity of real-world editing cases, leading to models with limited editing and generalization abilities.
Therefore, there is an urgent need for a high-quality dataset to facilitate real-world text-guided image editing. In this paper, we present MAGICBRUSH, a large-scale and manually annotated dataset for instruction-guided real image editing. We adopt natural language instruction [29, 4, 41, 22] for its flexibility, which enables users to easily express desired edits with phrases like "Remove the crowd in the background" or others shown in Figure 1. Additionally, we extend the dataset to include the multi-turn scenario considering the editing could be conducted iteratively on an image in practice.
We employ a rigorous training and selection for crowd workers, where they need to pass a qualification quiz and undergo manual grading during a trial period. Ongoing spot-checks ensure consistent quality, and failure to maintain high standards results in elimination from the task as shown in Figure 2. During the task, qualified workers need to propose edit instructions and utilize the DALL-E 2 [31] image editing platform to interactively synthesize target image. They will interact with the DALL-E 2 platform with different prompts and hyperparameters until they harvest their desired outputs, otherwise, the example will be dropped. Workers may perform continuous edits on the input image, leading to a series of edit turns. Each turn has a source image (may be the original or output from the previous turn), an instruction, and a target image. We refer to such a complete edit process on a real input image as an edit session. Eventually, we manually check the generated images to ensure quality. MAGICBRUSH consists of 5,313 sessions and 10,388 turns, supporting various editing scenarios including single-/multi-turn, mask-provided, and mask-free for both training and evaluation.
Experiments show that an end-to-end editing method InstructPix2Pix [4], delivers much better results after fine-tuning on MAGICBRUSH and outperforms other baselines according to human preferences. Furthermore, we conduct extensive experiments to evaluate current editing methods from multiple dimensions including quantitative, qualitative, and human evaluations. All these results reveal the challenging nature of MAGICBRUSH and the gap between existing methods and real-world editing needs, calling for more advanced model development in the future.
2.1 Text-guided Image Editing
Editing real images has long been an essential task in the field of image processing [27] and recent text-guided image editing has drawn considerable attention. Specifically, it can be categorized into three types in terms of different forms of text.
Global Description-guided Editing. Previous methods build fine-grained word and image region alignment for image editing [9, 17, 18]. Recently, Prompt2Prompt [14] modifies words in the original prompts to perform both local editing and global editing by cross-attention control. With the re
Table 1: Comparison of different image editing datasets. Flower and Bird are domain-specific datasets with global descriptions of target images. EditBench adopts masks (white regions) and local descriptions as guidance, and the size (240) may be insufficient for training. Due to the automatic synthesis process, InstructPix2Pix may contain failure cases.
Datasets	Real Image?	Open-domain?	Multi-turn?	# Edits	Source	Example Text	Target
Oxford-Flower [26]	✓	X	X	8,189	“numerous pale yellow petals and green pedicel with green oval leaves”
CUB-Bird [38]	✓	X	X	11,788	“this is a grey bird with a brown and yellow tail wing and a red head”
EditBench [37]	✓	✓	X	240	“a flat, dark-colored skateboard with yellow wheels”
InstructPix2Pix [4]	X	✓	X	313,010	“add a cat”
MAGICBRUSH	✓	✓	✓	10,388	“make the man ride a motorcycle”
weighting technique, follow-up work Null Text Inversion [24] further removes the need of original caption for editing by optimizing the inverted diffusion trajectory of the input image. Imagic [16] optimizes a text embedding that aligns with the input image, then interpolates it with the target description, thus generating correspondingly different images for editing. In addition, Text2LIVE [2] trains a model to add an edit layer and combines the edit layer and input image to enable local editing. For global description-guided editing, generally CLIP [30] can be applied to rank generated images w.r.t the alignment, thereby delivering higher-ranked results. However, the requirement for detailed descriptions of the target image poses an inconvenience for users.
Local Description-guided Editing. Another line of work utilizes masked regions and corresponding regional descriptions for local editing. Blended Diffusion [1] blends edited areas with the other parts of the image at different noise levels along the diffusion process. Imagen Editor [37] trains diffusion editing models by inpainting the masked objects. Local description-guided editing enables fine-grained control by using masks and preserves the other areas intact. However, this method places a greater burden on users, as they must provide additional masks. Also, this approach may be complicated for certain editing types, such as object removal due to the difficulty of describing missing elements.
Instruction-guided Editing. Another form of text is instruction, which describes which aspect and how an image should be edited, such as "change the season to spring". Instruction-guided editing, as initially proposed in various studies [11, 13, 42], enables users to edit images without requiring elaborate descriptions or region masking. With advancements in instruction following [29] and image synthesis [15], InstructPix2Pix [4] and SuTI [7] learn to edit images using instructions. Trained with synthetic texts by fine-tuned GPT-3 and images by Prompt2Prompt [14], InstructPix2Pix enables image editing by following instructions. Later work HIVE [41] introduces more training triplets and human ranking results to provide stronger supervision signals for better model training.
2.2 Image Editing Datasets
Table 1 compares various semantic editing datasets. Prior work [9, 40, 39, 17, 18] repurposes closed-domain image caption datasets [26, 38, 32] for image editing. However, these datasets primarily focus on specific categories like birds and flowers, resulting in limited generalization abilities for the models trained on them. In contrast, open-domain editing meets real-world needs, but high-quality data for training are scarce and challenging to obtain. Although large-scale silver data can be automatically synthesized [4], Table 1 shows the quality may not be desired. EditBench [37] is manually curated
Figure 2: The three-stage crowdsourcing workflow designed for dataset construction.
while it includes only 240 examples, which is insufficient for model training and comprehensive evaluations. Consequently, there is an urgent need for a manually annotated and large-scale dataset.
3 MAGICBRUSH Dataset
Instruction-guided image editing aims to edit a given image following the instruction. In terms of the editing guidance type, this task can be divided into two settings: In mask-free setting, given a source image  $I_{s}$  and a textual instruction  $T$  of how to edit this image, models are required to generate a target image  $I_{t}$  following the instruction. In mask-provided setting, models take an additional free-form mask  $M$  to limit the editing region, in addition to the source image and textual instruction. This setting is easier for models but less user-friendly as it requires extra guidance (mask) from users.
Orthogonally, depending on whether the edits are conducted iteratively, we can categorize instruction-guided image editing into two scenarios: single-turn and multi-turn. In multi-turn scenario, models take the source image  $I_{s}$  and a sequence of textual instructions  $\{T_{1}, T_{2}, \ldots, T_{n}\}$  to generate intermediate images  $\{\widehat{I_{t_1}}, \ldots, \widehat{I_{t_{n-1}}}\}$  and final image  $\widehat{I_{t_n}}$ . We term the entire process involving iterative edits as an edit session. The evaluation compares  $\widehat{I_{t_n}}$  with the ground truth final image  $I_{t_n}$ . In single-turn scenario, models take both the original source images and intermediate ground truth images  $\{I_s, I_{t_1}, \ldots, I_{t_{n-1}}\}$  as input, editing them only once with corresponding instructions to have  $\{\widetilde{I_{t_1}}, \widetilde{I_{t_2}}, \ldots, \widetilde{I_{t_n}}\}$ , respectively. Note that  $\widetilde{I_{t_i}}$  and  $\widetilde{I_{t_i}}$  are usually different except when  $i = 1$  where models take the same source image  $I_{s}$  and instruction  $T_{1}$ . For single-turn evaluation, we compare all generated images  $\{\widetilde{I_{t_1}}, \widetilde{I_{t_2}}, \ldots, \widetilde{I_{t_n}}\}$  and ground truths  $\{I_{t_1}, I_{t_2}, \ldots, I_{t_n}\}$  pairwise.
Among these scenarios, mask-free multi-turn editing is the most user-friendly yet challenging setting. Users can achieve complex editing goals with just textual instructions; however, this requires models to edit images iteratively, which easily leads to error accumulations.
3.2 Dataset Annotation Pipeline
We focus on real image editing and sample source images from MS COCO dataset [19] for subsequent annotations. We balance 80 object classes of COCO image to increase diversity, thus reducing the over-representation of the person object while keeping the image diversity. Figure 3a shows the final distribution of MAGICBRUSH, with  $34.0\%$  person-included images.
We hire crowd workers on Amazon Mechanical Turk (AMT) to manually annotate images using the DALL-E 2 platform. DALL-E 2 is a highly capable text-guided image synthesis platform that can generate high-quality candidate images for editing purposes. However, it requires expertise in providing specific editing guidance, including both global descriptions and masked regions. To
2AMT: https://www.mturk.com, DALL-E 2: https://openai.com/product/dall-e-2
(a) Top 20 object class distribution.
(b) Statistics of edit sessions and turns in each data split.
Figure 3: Statistics for the MAGICBRUSH dataset.
Number of	Train	Dev	Test	Overall
Edit Sessions	4,512	266	535	5,313
- Sessions with One Edit	1,789	100	216	2,105
- Sessions with Two Edits	1,151	70	120	1,341
- Sessions with Three Edits	1,572	96	199	1,867
Edit Turns	8,807	528	1,053	10,388
ensure the workers could proficiently use the DALL-E 2 platform, we provide them with detailed tutorials, teaching them how to edit images by writing prompts and drawing masks. We employ a stringent worker selection process as shown in Figure 2, and ultimately select 19 workers after thorough filtering. In recognition of the workers' contributions, we spend around $1 for each edit turn, which includes payment for workers on AMT along with the DALL-E 2 platform fees. Qualified workers will interact with DALL-E 2 using various prompts and masks until they achieve desired target images. Please refer to Appendix E for more annotation details.
Specifically, starting from the first edit turn, workers propose a textual instruction  $T_{1}$ , its corresponding global description, and a free-form region mask  $M_{1}$  to enable high-quality image synthesis. Then workers try to select the most description-faithful and photo-realistic synthesized image as target image. Note that workers may need to modify their descriptions and masks to find a qualified target image, or even restart with another instruction after several trials. After getting a qualified target image  $I_{t_1}$ , workers may repeat the annotation process with a new textual instruction  $T_{2}$  based on the current target image  $I_{t_1}$  to obtain  $I_{t_2}$ . In practice, we limit the max number of turns  $n$  to 3 for a session, considering workers' possible lack of motivation or inspiration for annotating more turns.
3.3 Dataset Analysis and Quality Evaluation
Data Composition. Through crowdsourcing, we collect a large-scale instruction-guided image editing dataset named MAGICBRUSH, consisting of over 5K edit sessions and more than 10K edit turns. Figure 3b provides the data splits, as well as the distributions of sessions with varying numbers of edits. Meanwhile, MAGICBRUSH includes a wide range of edit instructions such as object addition/replacement/removal, action changes, color alterations, text or pattern modifications, and object quantity adjustments. The keywords associated with each edit type demonstrate a broad spectrum, covering various objects, actions, and attributes as shown in Figure 4. This diversity indicates that MAGICBRUSH well captures a rich array of editing scenarios, allowing for comprehensive training and evaluation of instruction-guided image editing models.
Data Quality Evaluation. We invite five AMT workers to review 500 randomly sampled edit turns from MAGICBRUSH, with each evaluating 100 turns. Given an edit turn (source image, edit instruction, and target image), the worker is required to measure the edited image from two aspects: consistency and image quality. Consistency evaluates how well the editing to the original image aligns with the instruction. Image quality assesses the overall quality of the edited image, considering factors such as maintaining the visual fidelity of the original image, seamless blending of edited elements with the original image, and the natural appearance of the changes. Workers provide a score between 1 and 5 for each criterion. The average scores for consistency and image quality are reported as 4.1 and 3.9 out of 5.0, respectively. Compared to edited images by existing methods in Section 4.4, these numbers demonstrate the high quality of the MAGICBRUSH dataset.
Baselines. For comprehensiveness, we consider multiple baselines in both mask-free and mask-provided settings. For all baselines, we adopt the default hyperparameters available in the official
Figure 4: An overview of keywords in edit instructions. The inner circle depicts the types of edits and outer circle showcases the most frequent words used within each type.
code repositories to guarantee reproducibility and fairness. Given that some baselines may require global and local descriptions, inspired by InstructPix2Pix [4], we instruct ChatGPT [28] to generate desired text formats. Please refer to the Appendix C.4 for prompt details. Specifically, for mask-free editing baselines, we consider: (1) Open-Edit [21], (2) VQGAN-CLIP [8], (3) SD-SDEdit [23], (4) Text2LIVE [2], (5) Null Text Inversion [24], (6) InstructPix2Pix [4] and its fine-tuned version on the training set of MAGICBRUSH, (7) HIVE [41] and its fine-tuned version on MAGICBRUSH. For mask-provided baselines, we consider: (1) GLOWE [25] and (2) Blended Diffusion [1]. Please refer to Appendix C.2 for more implementation and fine-tuning details.
Evaluation Metrics. We utilize L1 and L2 to measure the average pixel-level absolute difference between the generated image and ground truth image. In addition, we adopt CLIP-I and DINO, which measure the image quality with the cosine similarity between the generated image and reference ground truth image using their CLIP [30] and DINO [6] embeddings. Finally, CLIP-T [34, 7] is used to measure the text-image alignment with the cosine similarity between local descriptions and generated images CLIP embeddings. We use local description because the global one is not specific to the editing region and the edit instruction may not describe the target image.
We evaluate mask-free and mask-provided baselines separately with the same 535 sessions from test set, as the latter requires mask as additional editing guidance, making it relatively easier. For each setting, we consider single- and multi-turn editing scenarios described in Section 3.1.
Mask-free Editing. Table 2 shows the results of mask-free methods which are given instructions only to edit images. We have the following observations: (1) In general, all methods perform worse in the multi-turn scenario due to the error accumulation in iterative editing. (2) The off-the-shelf InstructPix2Pix [4] checkpoint is not competitive compared to other baselines, in both single-turn and multi-turn scenarios. However, after fine-tuning on MAGICBRUSH, InstructPix2Pix shows significant performance improvements across all metrics, achieving the best or second-best results under most metrics. Such improvement introduced by MAGICBRUSH is consistent on HIVE [41]. These suggest that instruction-guided image editing models could substantially benefit from training on our MAGICBRUSH dataset, demonstrating its usefulness. (3) Text2LIVE [2] performs well in L1
Table 2: Quantitative study on mask-free baselines on MAGICBRUSH test set. Multi-turn setting evaluates the final target images that iteratively edited on the first source images in edit sessions. The best results are marked in bold.
Settings	Methods	L1↓	L2↓	CLIP-I↑	DINO↑	CLIP-T↑
Single-turn	Global Description-guided
Open-Edit [21]	0.1430	0.0431	0.8381	0.7632	0.2610
VQGAN-CLIP [8]	0.2200	0.0833	0.6751	0.4946	0.3879
SD-SDEdit [23]	0.1014	0.0278	0.8526	0.7726	0.2777
Text2LIVE [2]	0.0636	0.0169	0.9244	0.8807	0.2424
Null Text Inversion [24]	0.0749	0.0197	0.8827	0.8206	0.2737
Instruction-guided
HIVE [41]	0.1092	0.0341	0.8519	0.7500	0.2752
w/ MagicBrush	0.0658	0.0224	0.9189	0.8655	0.2812
InstructPix2Pix [4]	0.1122	0.0371	0.8524	0.7428	0.2764
w/ MagicBrush	0.0625	0.0203	0.9332	0.8987	0.2781
Multi-turn	Global Description-guided
Open-Edit [21]	0.1655	0.0550	0.8038	0.6835	0.2527
VQGAN-CLIP [8]	0.2471	0.1025	0.6606	0.4592	0.3845
SD-SDEdit [23]	0.1616	0.0602	0.7933	0.6212	0.2694
Text2LIVE [2]	0.0989	0.0284	0.8795	0.7926	0.2716
Null Text Inversion [24]	0.1057	0.0335	0.8468	0.7529	0.2710
Instruction-guided
HIVE [41]	0.1521	0.0557	0.8004	0.6463	0.2673
w/ MagicBrush	0.0966	0.0365	0.8785	0.7891	0.2796
InstructPix2Pix [4]	0.1584	0.0598	0.7924	0.6177	0.2726
w/ MagicBrush	0.0964	0.0353	0.8924	0.8273	0.2754
Table 3: Quantitative study on mask-provided baselines on MAGICBRUSH test set. L1, L2, and CLIP-T are measured over the masked regions only. The best results are marked in bold.
Settings	Methods	L1↓	L2↓	CLIP-I↑	DINO↑	CLIP-T↑
Single-turn	GLIDE [25]	3.4973	115.8347	0.9487	0.9206	0.2249
Blended Diffusion [1]	3.5631	119.2813	0.9291	0.8644	0.2622
Multi-turn	GLIDE [25]	11.7487	1079.5997	0.9094	0.8494	0.2252
Blended Diffusion [1]	14.5439	1510.2271	0.8782	0.7690	0.2619
and L2 evaluations, likely due to the addition of an extra editing layer that minimizes changes to the source image. As a result, edited images fail to satisfy the instructions, as evidenced by the low CLIP-T score. VQGAN-CLIP [8] achieves the highest CLIP-T score because it fine-tunes the model during inference with CLIP as the direct supervision. However, the edited images may change too significantly, leading to unfavorable results on other metrics.
Mask-provided Editing. Table 3 lists the results of two mask-provided methods. As observed in the mask-free setting, the multi-turn scenario is more challenging than the single-turn scenario. While both mask-provided methods achieve high scores under the CLIP-I and DINO metrics, they fail to deliver satisfactory results according to the other three metrics (L1, L2, and CLIP-T) that evaluate local regions. Notably, after tuning on MAGICBRUSH, InstructPix2Pix [4] achieves better editing results than mask-provided Blended Diffusion [1] in terms of CLIP-I and DINO metrics. This suggests that fine-tuning with our data could maintain good image quality.
We present the results of the top-performing mask-free (Text2LIVE [2]) and mask-provided (GLIDE [25]) methods in our qualitative analysis. We also compare the original and fine-tuned checkpoints of InstructPix2Pix [4]. Figure 5 illustrates the iterative results of these four models
Figure 5: Qualitative evaluation of multi-turn editing scenario. We provide all baselines their desired input formats (e.g., masks and local descriptions for GLIDE).
and ground truth images from MAGICBRUSH. Both Text2LIVE and GLIDE are unsuccessful in editing the man's face and clothes. The original InstructPix2Pix changes the images following the instructions; however, the resulting images exhibit excessive modification and lack photorealism. Fine-tuning InstructPix2Pix on MAGICBRUSH alleviates this issue, but the images remain notably inferior to the ground truth ones. Please see Appendix D for more examples of qualitative evaluation.
4.4 Human Evaluation
We conduct comprehensive human evaluations to assess both consistency and image quality on generated images. Our evaluations encompass three tasks: multi-choice image comparison, one-on-one comparison, and individual image evaluation. We randomly sample 100 image examples from test set for each task and hire 5 AMT workers as evaluators to perform the tasks. For each task, the images are evenly assigned to evaluators and the averaged scores (if applicable) are reported.