SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Multi-choice Comparison. The multi-choice comparison involves four top-performing methods in Table 2 and Table 3, including Text2LIVE, GLIDE, InstructPix2Pix, and fine-tuned InstructPix2Pix on MAGICBRUSH. For each example, evaluators need to select the best edited image based on consistency and image quality, respectively. The results in Table 4 indicate that fine-tuned InstructPix2Pix attains the highest performance, significantly surpassing the other three methods. This outcome validates the effectiveness of training on our MAGICBRUSH dataset. Interestingly, while Text2LIVE achieves a high score in auto evaluation, its performance in human evaluation appears to be less desirable, especially in terms of the instruction consistency. This indicates current automatic metrics that focus on the overall image quality may not align well with human preferences, emphasizing the need for future research to develop better automatic metrics.
Table 4: Multi-choice comparison of four methods. The numbers represent the frequency of each method being chosen as the best for each aspect.
Text2LIVE [2]	GLIDE [25]	InstructPix2Pix [4]	Fine-tuned InstructPix2Pix
Consistency	0	16	33	51
Image Quality	9	15	27	49
Table 5: One-on-one comparisons between fine-tuned InstructPix2Pix and other methods including InstructPix2Pix and Text2LIVE, as well as ground truth (GT). The numbers in the table indicate the frequency of each method being chosen as the better option. To account for scenarios where two methods perform equally, we include a "Tie" option in each question for comprehensive evaluation.
Settings	Consistency	Image Quality
Single-turn	Fine-tuned InstructPix2Pix 40	InstructPix2Pix [4]	Tie	Fine-tuned InstructPix2Pix 48	InstructPix2Pix [4]	Tie
35	25	33	19
Fine-tuned InstructPix2Pix 68	Text2LIVE [2]	Tie	Fine-tuned InstructPix2Pix 61	Text2LIVE [2]	Tie
4	28	19	20
Multi-turn	Fine-tuned InstructPix2Pix 13	GT (Turn 1)	Tie	Fine-tuned InstructPix2Pix 19	GT (Turn 1)	Tie
72	15	64	17
Fine-tuned InstructPix2Pix 13	GT (Turn 2)	Tie	Fine-tuned InstructPix2Pix 19	GT (Turn 2)	Tie
80	7	60	21
Fine-tuned InstructPix2Pix 11	GT (Turn 3)	Tie	Fine-tuned InstructPix2Pix 6	GT (Turn 3)	Tie
80	9	75	19
One-on-one Comparison. The one-on-one comparison provides a detailed and nuanced evaluation of the fine-tuned InstructPix2Pix by comparing it against strong baselines and ground truth. Evaluators are asked to determine the preferred option based on consistency and image quality, respectively. We divide the comparisons into two scenarios as mentioned in Section 3.1: (1) In the single-turn scenario, we compare fine-tuned InstructPix2Pix and two other methods (InstructPix2Pix and Text2LIVE). As shown in Table 5, fine-tuned InstructPix2Pix consistently outperforms the other two methods in terms of both consistency and image quality. (2) In the multi-turn scenario, we compare the fine-tuned InstructPix2Pix with ground truth images to observe how the quality of edited images varies across different turns. The results reveal that the performance gap generally widens as the number of edit turn increases. This finding highlights the challenges associated with error accumulation in current top-performing models and underscores the difficulties posed by our dataset.
Individual Evaluation. The individual evaluation employs a 5-point Likert scale to measure the quality of individual images generated by four specific models, gathering subjective user feedback. Evaluators are asked to rate the images on a scale from 1 to 5, assessing both consistency and image quality. Each evaluator receives an equal share of the images, specifically evaluating 80 images in total, with 20 images from each of the four models. The results in Table 6 clearly demonstrate that fine-tuned InstructPix2Pix outperforms Text2LIVE and GLIDE, and further improves upon the performance of InstructPix2Pix. This finding highlights the advantages of training or fine-tuning models using the MAGICBRUSH dataset.
Table 6: Individual evaluation using a 5-point Likert scale. The numbers in the table represent the average scores calculated for each aspect.
Consistency	Image Quality
Text2LIVE [2]	1.1	2.8
GLIDE [25]	1.8	2.8
InstructPix2Pix [4]	3.0	3.2
Fine-tuned InstructPix2Pix	3.1	3.6
In this work, we present MAGICBRUSH, a large-scale and manually annotated dataset for instruction-guided real image editing. Although extensive experiments show that InstructPix2Pix fine-tuned on
MAGICBRUSH achieves the best results, its edited images are still notably inferior compared to the ground truth ones. This observation indicates the effectiveness of our dataset for training and the gap between current methods and real-world editing needs. We hope MAGICBRUSH will contribute to the development of more advanced models and human-preference-aligned evaluation metrics for instruction-guided real image editing in the future.
The authors would like to thank colleagues from the OSU NLP group for their constructive feedback, Yuxuan Sun for discussing the fine-tuning of InstructPix2Pix, and the contributors from the Amazon Mechanical Turk platform for their participation in the study and assistance with data collection. This research was sponsored in part by NSF CAREER #1942980, ARL W911NF2220144, NSF OAC 2112606, and NSF OAC 2118240. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.
[1] O. Avrahami, D. Lischinski, and O. Fried. Blended diffusion for text-driven editing of natural images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 18187-18197. IEEE, 2022. URL https://doi.org/10.1109/CVPR52688.2022.01767.
[2] O. Bar-Tal, D. Ofri-Amar, R. Fridman, Y. Kasten, and T. Dekel. Text2live: Text-driven layered image and video editing. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, volume 13675 of Lecture Notes in Computer Science, pages 707-723. Springer, 2022. URL https://doi.org/10.1007/978-3-031-19784-0_41.
[3] D. Bau, A. Andonian, A. Cui, Y. Park, A. Jahanian, A. Oliva, and A. Torralba. Paint by word. CoRR, abs/2103.10951, 2021. URL https://arxiv.org/abs/2103.10951.
[4] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. CoRR, abs/2211.09800, 2022. URL https://doi.org/10.48550/arXiv.2211.09800.
[5] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/bit/1457c0d6bfcb4967418bfb8ac142f64a-AAbstract.html.
[6] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[7] W. Chen, H. Hu, Y. Li, N. Ruiz, X. Jia, M.-W. Chang, and W. W. Cohen. Subject-driven text-to-image generation via apprenticeship learning, 2023.
[8] K. Crowson, S. Biderman, D. Kornis, D. Stander, E. Hallahan, L. Castricato, and E. Raff. VQGAN-CLIP: open domain image generation and editing with natural language guidance. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, volume 13697 of Lecture Notes in Computer Science, pages 88-105. Springer, 2022. URL https://doi.org/10.1007/978-3-031-19836-6_6.
[34] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22500-22510, June 2023.
[35] Y. Shi, X. Yang, Y. Wan, and X. Shen. Semanticstylegan: Learning compositional generative priors for controllable image synthesis and editing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11244-11254. IEEE, 2022. URL https://doi.org/10.1109/CVPR52688.2022.01097.
[36] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=St1giarCHLP.
[37] S. Wang, C. Sahara, C. Montgomery, J. Pont-Tuset, S. Noy, S. Pellegrini, Y. Onoe, S. Laszlo, D. J. Fleet, R. Soricut, J. Baldridge, M. Norouzi, P. Anderson, and W. Chan. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. CoRR, abs/2212.06909, 2022. URL https://doi.org/10.48550/arXiv.2212.06909.
[38] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-ucsd birds 200. Technical Report CNS-TR-201, Caltech, 2010. URL /se3/wp-content/uploads/2014/09/WelinderEetal10_CUB-200.pdf, http://www.vision.caltech.edu/visipedia/CUB-200.html.
[39] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 1316-1324. Computer Vision Foundation / IEEE Computer Society, 2018. URL http://openaccess.thecvf.com/content_cvpr_2018/html/Xu_AttnGAN_Fine-Grained_Text_CVPR_2018_paper.html.
[40] H. Zhang, T. Xu, and H. Li. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5908-5916. IEEE Computer Society, 2017. URL https://doi.org/10.1109/ICCV.2017.629.
[41] S. Zhang, X. Yang, Y. Feng, C. Qin, C. Chen, N. Yu, Z. Chen, H. Wang, S. Savarese, S. Ermon, C. Xiong, and R. Xu. HIVE: harnessing human feedback for instructional visual editing. CoRR, abs/2303.09618, 2023. URL https://doi.org/10.48550/arXiv.2303.09618.
[42] T. Zhang, H. Tseng, L. Jiang, W. Yang, H. Lee, and I. Essa. Text as neural operator: Image manipulation by text instruction. In MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, pages 1893-1902. ACM, 2021. URL https://doi.org/10.1145/3474085.3475343.
Our supplementary includes the following sections:
- Section B: Discussions. Discussions of Limitations, Alleviating Potential Model Bias, Social Impacts, Ethical Considerations, and License of Assets.
- Section C: Implementation Details. Details for implementing baselines and fine-tuning InstructPix2Pix with MAGICBRUSH.
- Section D: More Qualitative Study. More qualitative study including both single-turn and multi-turn scenarios.
- Section E: Data Annotation. Details for dataset collection and image quality evaluation.
We share the following artifacts:
Table 7: Shared artifacts in this work, we protect the test split with a password to avoid web crawling for model training.
Artifact	Link	License
Homepage	https://osu-nlp-group.github.io/MagicBrush/	-
Code Repository	https://github.com/OSU-NLP-Group/MagicBrush	CC BY 4.0
Training and Dev Set	https://huggingface.co/datasets/osunlp/MagicBrush	CC BY 4.0
Test Set	https://shorturl.at/alHMO (Password: MagicBrush)	CC BY 4.0
B Discussions
Although our image annotation is based on a lot of manual effort and conducted on the powerful editing platform (DALL-E 2), a small portion of edits  $(<5\%)$  may contain minor extra modifications that are not mentioned by the instruction or may still look slightly unnatural to some individuals. That being said, we believe it would not affect the overall quality of MAGICBRUSH as the experiments have shown that our dataset can largely enhance the model's abilities of editing real images w.r.t the given instruction.
While MAGICBRUSH supports various edit types on real images, it does not contain data for global editing (e.g., style transfer) due to annotation built upon DALL-E 2. However, such edit turn could be easily obtained automatically [4] due to its less photorealism and more artistic nature.
B.2 Alleviating the Potential Model Bias
After conducting an in-depth pilot exploration on various generative models, including commercial image-editing platforms, we have found that DALL-E 2 is one of the best available editing models. It is highly likely that users can obtain satisfactory images that meet their editing goals, provided they carry out sufficient trials on the prompting and masking. However, solely using one model for ground truth generation may result in the potential bias inherent in that model.
To alleviate this, we adopt the following strategies from two aspects: 1) Diversity of instruction: Through clear guidance in our tutorial and frequent communication via email, we strongly encourage workers to design diverse instruction. In practice, we reject some repetitive or trivial edits and suggest alternatives to ensure the diversity. 2) Diversity of images: We carefully design a sampling strategy to ensure the objects in the images are more balanced and decrease the chance of sampling simple images with fewer objects. In this way, the editing largely varies since the edited regions are required to be naturally blended with the context. With these efforts, MAGICBRUSH has less recurring edit patterns and higher diversity, thus minimizing potential biases.
That being said, admittedly, it is challenging to eliminate the inherent biases completely. We commit to remaining alert for potential biases in our dataset identified by the community, and will take prompt rectification actions.
B.3 Social Impacts
MAGICBRUSH has the potential to significantly improve the capabilities of text-guided image editing systems, enabling a broader range of users to easily manipulate images. On one hand, this could lead to numerous positive social impacts: users can achieve their editing goals through instructions alone, without the need for professional editing knowledge, such as using Photoshop or painting. Such an effortless editing process can save users' time spent on manual operation, resulting in increased efficiency. Furthermore, it can facilitate image creation and manipulation for users with visual or motor impairments, given they can rely on language instructions as input.
On the other hand, the potential risks associated with such advanced image editing systems deserve attention. Malicious users could exploit editing tools to create realistic fake or harmful content, leading to the spread of misinformation. It is essential to implement appropriate safeguards and responsible AI frameworks when developing user-friendly image editing systems.
B.4 Ethical Considerations
The COCO [19] dataset focuses on common objects and context, rather than specific people or places. In our annotation guidelines, we also forbid annotators from creating any identifiable information (e.g., human faces). Furthermore, DALL-E 2 adheres to strict rules to exclude prompts related to harmful, inappropriate, or sensitive content. As a result, MAGICBRUSH inherently minimizes the potential for privacy or harmful concerns as it relies on images sourced from the COCO dataset and annotations built upon DALL-E 2.
To ensure the collection of high-quality data and fair treatment of our crowdworkers, we have implemented a meticulous payment plan for the AMT task. We conduct a pilot study to estimate the
average time required to complete a session. It reveals that the duration ranges from 4 to 8 minutes, depending on the number of edit turns performed by the workers within each session. This results in a total annotation time of approximately 529 worker hours. This information also allows us to appropriately adjust the payment, ensuring it exceeds the minimum wage amount in our state. As a result, we offer an initial payment of 80 cents for the first edit turn in each session, along with a bonus of 40 cents for each additional edit turn within the same session. This allows workers to potentially earn up to  $1.6 per session, encouraging their active participation and rewarding their efforts accordingly. In total, the cost of creating the MAGICBRUSH dataset amounts to approximately$ 11,000 which includes the payments made on AMT ($8,000) and DALL-E 2 API ($3,000) costs.
B.5 License of Assets
For baselines, VQGAN-CLIP [8], Text2LIVE [2], and Blended Diffusion [1] are under the MIT License. SD-SDEdit [33, 23] is released under the Creative ML OpenRAIL-M License, and InstructPix2Pix [4] inherits this license as it is built upon Stable Diffusion. Null Text Inversion [24] and GLIDE [25] are under the Apache-2.0 License.
For dataset, COCO [19] is under Creative Commons Attribution 4.0 License. According to DALL-E 2, we own the images created with DALL-E 2, including the right to reprint, sell, and merchandise. We decide to release MAGICBRUSH under Creative Commons Attribution 4.0 License for easy access in the research community. The license allows users to share and adapt the dataset for any purpose, even commercially, as long as appropriate credit is given and any changes made are indicated. By providing the dataset under this license, we hope to encourage researchers and practitioners to explore and advance the field of text-guided image editing further.
C.1 COCO Image Sampling
Given the highly unbalanced distribution of objects in COCO, where  $54.2\%$  of images contain a person, we employ a class-balanced sampling strategy for the 80 classes. In particular, for each class, we select one image containing an object from the target class, ensuring it has no overlap with the current image pool. This process is repeated as we move through each class. Notably, one COCO image may contain multiple objects from different classes, so it is possible to sample images with a person for non-person classes. To mitigate the over-representation of person class, we prioritize selecting images without a person for non-person classes by reducing the sampling probability of images containing a person by half.
C.2 Baseline Details
For all baselines, we adopt the default hyperparameters available in the official code repositories to guarantee reproducibility and fairness. Specifically, for mask-free editing baselines, we consider:
(1) Open-Edit [21] is a GAN-based method pre-trained with reconstruction loss and fine-tuned on the given image with consistency loss. It edits image by performing arithmetic operations on word embeddings within a shared vector space with visual features.
(2) VQGAN-CLIP [8] fine-tunes VQGAN [12] with CLIP embedding [30] similarity between generated image and target text. Then it generates the image with the optimized VQGAN embedding.
(3) SD-SDEdit [23] is a tuning-free method built upon Stable Diffusion [33]. Based on the target description, SDEdit adds stochastic differential equation noise to the source image and then denoises the target image through that prior.
(4) Text2LIVE [2] fine-tunes Vision Transformer [10] to generate the edited object on the extra edited layer with data augmentation and CLIP [30] supervision. The target image is the composite of the extra edit layer and the original layer.
(5) Null Text Inversion [24] optimizes DDIM [36] trajectory to restore the source image and then performs image editing on the denoising process with text-image cross-attention control [14].
(6) InstructPix2Pix [4] is pre-trained with automatically curated instruction-following editing data, initialized from Stable Diffusion [33]. It edits the source image by controlling the faithfulness to instruction and similarity with the source image, without any test-time tuning.
(7) HIVE [41] is trained with more data synthesized using a method similar to InstructPix2Pix [4] and is further fine-tuned with a reward model trained with human-ranked data.
For mask-provided baselines, we consider:
(1) GLIDE [25] is trained with 67M text-image pairs where all images are person-free. To edit, it fills in the masked region of an image conditioned on the local description with CLIP [30] guidance.
(2) Blended Diffusion [1] resorts to CLIP [30] guidance during a masked region denoising process and blends it with the context in the noisy source image at each denoising timestep to increase the region-context consistency of the generated target image.
C.3 InstructPix2Pix Fine-tuning Details.
We continually fine-tune the checkpoint with the training set of MAGICBRUSH. Specifically, we train 168 epochs on  $2 \times 40$ GB NVIDIA A100 GPUs with a total batch size of 64. Following prior work [4], we use a  $256 \times 256$  image resolution and the same training strategies and hyper-parameters.
C.4 ChatGPT Prompts
Table 8: Prompts on ChatGPT for global and local description generation.
Global Description	Given the original caption and a edit instruction, write a caption after editing.
Original Caption: Painting of The Flying Scotsman train at York station Edit Instruction: add airplane wings Final Caption: Painting of The Flying Scotsman train with airplane wings at York station
Original Caption: Old Boat at Sunderland Point by Steve Liptrot Edit Instruction: remove the boat Final Caption: Empty Sunderland Point by Steve Liptrot
Original Caption: "Charles Lindbergh""Spirit of St. Louis"" Edit Instruction: have it be about Beijing Final Caption: "Charles Lindbergh""Spirit of Beijing""
Original Caption: [CAPTION] Edit Instruction: [INSTRUCTION] Final Caption:
Local Description	Given the original caption and an edit instruction, write a local short description for specific location to describe the object. If it's removing, leave it blank.
Original Caption: Painting of The Flying Scotsman train at York station Edit Instruction: add airplane wings Local Caption: airplane wings
Original Caption: Old Boat at Sunderland Point by Steve Liptrot Edit Instruction: remove the boat Local Caption:
Original Caption: A demonic looking chucky like doll standing next to a white clock. Edit Instruction: Make the doll wear a hat Local Caption: hat
Original Caption: [CAPTION] Edit Instruction: [INSTRUCTION] Local Caption:
To transform the edit instruction to global description and local description required by other baselines and facilitate future research. Inspired by InstructPix2Pix [4], we instruct ChatGPT (davinci-turbo0301) to generate the target text formats given the input image caption and instruction. Specifically, as shown in Tab 8, we provide clear instructions and three in-context learning examples [5] for ChatGPT to learn the generation rules, thus generating the desired text formats for baselines.
D More Qualitative Study
Figure 7 shows the results of top-performing baselines in multi-turn editing scenarios. And the observation is consistent with that shown in Figure 5.
In addition, we show more baselines in the single-turn editing scenario in Figure 6. Even in such a relatively easier scenario, most baselines fail to edit precisely. Although InstructPix2Pix edits the images following the instruction to some extent, it tends to modify the images too much, resulting in the loss of some important details or incorrect changes.
Figure 6: Qualitative evaluation of single-turn editing scenario. We provide all baselines their desired input formats (e.g., masks and local descriptions for Blended Diffusion and GLIDE).
Figure 7: Qualitative evaluation of multi-turn editing scenario. We provide all baselines their desired input formats (e.g., masks and local descriptions for GLIDE).
E Data Annotation
Figure 8: Illustration of the step-by-step instructions in annotation tutorial.
E.1 Annotation Tutorial