Learning to Follow Object-Centric Image Editing Instructions Faithfully
Authors: Tuhin Chakrabarty
Kanishk Singh
Arkadiy Saakyan
Smaranda Muresan
Venue: EMNLP 2023 Findings (2023)
Keywords: natural language instruction
image editing
stable diffusion
diffusion model
text-to-image
multimodal
TL;DR: We improve the ability of text-to-image models to follow edit instructions.

Abstract: Natural language instructions are a powerful interface for editing the outputs of text-to-image diffusion models. However, several challenges need to be addressed: 1) underspecification (the need to model the implicit meaning of instructions) 2) grounding (the need to localize where the edit has to be performed), 3) faithfulness (the need to preserve the elements of the image not affected by the edit instruction). Current approaches focusing on image editing with natural language instructions rely on automatically generated paired data, which, as shown in our investigation, is noisy and sometimes nonsensical, exacerbating the above issues. Building on recent advances in segmentation, Chain-of-Thought prompting, and visual question answering, we significantly improve the quality of the paired data. In addition, we enhance the supervision signal by highlighting parts of the image that need to be changed by the instruction. The model fine-tuned on the improved data is capable of performing fine-grained object-centric edits better than state-of-the-art baselines, mitigating the problems outlined above, as shown by automatic and human evaluations. Moreover, our model is capable of generalizing to domains unseen during training, such as visual metaphors.

Learning to Follow Object-Centric Image Editing Instructions Faithfully
Tuhin Chakrabarty 1x Kanishk Singh' 1*
Arkadiy Saakyan' Smaranda Muresan 1.2 DDepartment of Computer Science, Columbia University
Data Science Institute, Columbia University
tuhin. chakr@cscrrsscccmmm www.rrccccoo.mmiii.............e wwwwwoww..c...oe.ooo...........l ks4018@columbia. edu, a. ..saakynnccs.comm cowumbia..aoomomooooo..o........ edu, smara@columbia. edu
Abstract
Natural language instructions are a powerful in- terface for editing the outputs of text-to-image diffusion models. However, several challenges need to be addressed: 1) underspecification (the need model the implicit meaning of in- structions) 2) grounding (the need to localize where the edit has to performed), 3) faith- fulness (the need to preserve the elements of the image not affected by the edit instruction). Current approaches focusing on image editing with natural language instructions rely au- tomatically generated paired data, which, as shown in our investigation, is noisy and some- times nonsensical, exacerbating the above is- sues. Building on recent advances segmenta- tion, Mhain-of-Thouue question answering, we significantly improve the quality of the paired data. In addition, we enhance the supervision signal by highlighting parts of the image that need to be changed by the instruction. The model fine-tuned on the improved data is capable of performing fine- grained object-centric edits better than state-of- the-art baselines, mitigating the problems out- lined above, as shown by automatic and human evaluations. Moreover, our model is capable of generalizing to domains unseen during training, such as visual metaphors.
(
Figure I: (a) Input Image (h) Edited image from In- structPix2Pix (Brooks et al., 2023) with the instruction Add lighthouse.
tools such as Adobe Photoshop', bridging the gap between technology and content creators.
prompting, and visual
While natural language instructions act as a powerful interface for editing images, following them remains a challenging task for several reasons. First, these instructions are often underspecified. requiring models to uncover their implicit meaning. For example, in Figure for the input image (a), the user provides a prompt Add lighthouse. The model needs to understand how lighthouse looks, that only one lighthouse needs to be added and that needs to be placed on land and not in the water. Second, models must be able to localize where the "background" is in the image so that the lighthouse can be added appropriately (grounding). Finally, models must follow instructions faithfully, i.e. pre- serve the elements of the image not affected by the edit instruction (e.g., both houses in Figure (a)). One of the key challenges for making progress on the task of image editing via natural language instructions the lack of high-quality annotated or naturally occurring data. Recently, Brooks et al. (2023) proposed a way to handle this by automat- ically creating paired dataset utilizing large lan- guage models and text-to-image models. They
1 Introduction
Frameworks for large-scale text-conditional image synthesis, which rely on diffusion processes (Sa- haria et al., 2022; Rombach et al., 2022a; Ramesh al., 2021; Nichol et al., 2021) have shown im- pressive generative capabilities and practical uses. Notably, image editing guided by text has garnered considerable attention due its ease use and scemingly high-quality results (Avrahami et al., 2022b,a; Hertz et al., 2022a; Kawar et al., 2023). These advances are now utilized in leading industry
Equal Contribution
badobe.com/sennee//geeee

---

(a) Add a lighthouse
(h)
(c)
(d)
Figure 2: Steps to create parallel data: (a) Input Image Edit Instruction: (b) Image with grounding the form of bounding box for the entity where transformation has to be made; (c) Masked localized segment of the grounded image where the transformation has be made; (d) Final output.
further train conditional diffusion model on this paired data of synthetic examples and show that their model is capable of editing images condi- tioned on natural language instructions at run-time However, closer inspection reveals that the data is noisy and sometimes nonsensical. As can be seen in Figure (b) the model adds three lighthouses instead of one, likely due to the underspecification of the instruction and the lack of grounding. Fur- thermore, it compromises faithfulness by removing both houses from the input image.
ages remain faithful to the entities in the original image. For this, generate relevant questions with 'Yes/No' answers regarding the unmodified elements in the image such as "Does this image contain cottage" or questions regarding objects in the instruction "Is there a lighthouse in the im- age" using the Vicuna-1 model (Chiang et al., 2023). Given an image-question pair, we then use the BLIP-2 VQA-model Li et al. (2023) to collect responses and re-rank generated images from Sta- ble Diffusion+Inpaintigg by faithfulness scores in terms of correct answers and select the best one for higher quality. We then fine-tune the model on this newly created parallel data. In addition, both during finetuning and inference, we enhance the supervision signal by denoising parts of the image that need to be changed by the instruction as shown in Figure 5. To summarize, contributions are:
To tackle these challenges we first create high- quality corpus starting from the parallel data re- leased by Brooks al. (2023). For example, given the caption of the input image in Figure Ocean Cottage by Todd Baxter, and an edit instruction Add a lighthouse, we use recent advances in Chain- of of-Thought prompting (Wei et al., 2022) to iden- tify whether the transformation can be performed in the context of the input image and what entity should be transformed. The entity and the input image are fed to object detection model Ground- ingDINO (Liu et al., 2023). As can be seen in Fig- ure 2 (b), DINO draws bounding box to denote "background". However, grounding in the form bounding boxes cannot entirely disentangle the entity of interest. Thus, we use recent advances in image segmentation (Kirillov et al., 2023) to identify the exact segment containing the entity that needs be masked (see Figure 2 (c)) and and (Rombach al.,
Improving the quality of existing paired datasets used for image editing via natural language in- structions with the help of recent advances in segmentation, Chain-of- Thought prompting, and visual question answering.
Curating test set of diverse non-noisy instruc- tions, consisting of both in-domain and out-of- domain examples and conducting thorough evaluation across SOTA baselines and our model ablations.
Demonstrating that fine-tuning a diffusion model our parallel data enhanced with supervision signal leads a significant improvement over several compelling baselines in terms of faithful- ness using TIFA scores (Hu et al., 2023) as well as instruction satisfiability. Our human evalua- tion corroborates these findings.
Stable inffuusion+Inainntii 2022a) with the masked image (c) to obtain the edited output (Figure 2(d)).
To further account for faithfulness in our paired training data we leverage techniques from VQA (Antol et al., 2015) to ensure that the edited im-

---

aa)
(b)
(c)
(d)
Figure 3: Examples of noisy edit instructions and image-pairs: (a) Make her look like an android; (b) Make the rocky mountains look like chessboard: (c) Replace her with bird; (d) Have it be lighthouse.
We release our code, data and pretrained models.?
quality of the training set and leads to non-faithful edits by fine-tuned diffusion models.
2 Data
2.2
High-Quality Training Dataset Curation
2.1 Problems with Existing Datasets
We describe our approach for curating the dataset with focus on addressing the above-mentioned challenges including underspecification, ground- ing, and faithfulness.
The dataset introduced by (Brooks et al., 2023) marked a significant step towards enabling diffu- sion models to comprehend instructions for im- age editing. However, since GPT-3 was used to generate captions and edit instructions (Figure in Appendix A), the constructed image-edit pairs suffered from various limitations that makes the editing process less precise and efficient.
Filtering noise and handling under-specification To improve the noisy synthetically generated edit instructions from (Brooks et al., 2023), leverage the reasoning capabilities of large language models through Chain-of- Thought prompting (Wei et al., 2022) to jointly predict whether the instruction is appropriate w.r.t the context present in the original caption and to generate the entity/region that needs to be grounded and changed for performing the edit, Table 3 in Appendix A shows how jointly elicit the edit entity well as a verdict on whether the transformation is possible. Large language models can uncover implicit entities image editing tasks based on their own commonsense knowledge, even without explicit mentions in the input caption or edit instruction. For instance, in Figure 4 given the original image with the caption Buttermere Lake District and edit instruction Add aurora borealis. the model outputs that the entity which the edit is applied sky, using the implicit commonsense knowledge that "aurora borealis" has to appear the sky.
One of the frequent concerns in the edit instruc- tions is that many of them are vague and incompre- hensible to perform a successful edit. Furthermore, many of the edit instructions suffer from language model hallucinations. For instance, in Figure (b), the instruction "Make the rocky mountains look like chessboard" is not sensible in the context of the input image. In Figure (c), the gold image does not actually represent the instruction, creating issues the instruction following the fine-tuned model. Furthermore, object-centric instructions are inherently harder for diffusion models. Models that cannot localize the corresponding entity or region in the image end up performing incorrect exces- sive modifications, resulting in images that are not faithful to the instruction input image can be seen in Figures 3 (c) and (d). This undermines the
Egithub. wtt.///t/tttceeccc www..//tiiiniccceeaaafauuuuidiaa
We filter object-centric image- mnage-instruction pairs

---

CAPTION:
Buttermere Lake District
Discard Sample
the edit possible?
NO
Grounding DTNO
PRONPT
YES
xdit Anti by:
INSTROCTION:
Add an aurora borealis
Segment Anything Mask
VQA there lake distfict in the painling"". DDos the image contain an auroma boraslis?
Stable Diffusion Inpainting
Figure 4: Steps create high-quality training parallel data: Given input image, caption, and edit instruction first use Chain-of- Thain-of-Thought (CoT) prompting with ChatGPT to identify whether the edit instruction is sensible and if it is, what the entity that needs be transformed. Using the LLM-generated edit entity we use GroundingDINO to localize it and SAM (Segment Anything Mask) to segment it. We then use Stable Diffusion Inpainting to generate 3 images and filter out the best image with the help of VQA.
that align with the original caption, discarding im- ages lacking specific segment for editing. This ensures high-quality, precise, and contextually ap- propriate entity-centric image editing.
text-to-image diffusion model capable of gener- ating photo-realistic images given any text input, with the extra capability of inpainting a specified area of an image given mask. However, this model does not understand implicit instructions and requires captions of the original image, so we use the "edited caption" (See Figure 9 in Appendix A) present in the existing dataset to perform the edit operation on the image. We feed the binary menge--egmentttion mask generated by the SAM model as seen in Figure 4 along with the input im- age and the edited caption to perform the required entity-centre edit on the image. Stable Diffusion Inpainting generated images might not be always faithful, so we generate three images per input and re-rank them based on faithfulness as described below.
Incorporating Grounding We utilize object detection and segmentation to provide addi- tional supervision signals for grounded- -mnaae edit- ing. The edit entity generated from ChatGPT (gpt-3.5-turbo) in the previous step is given as input to the open-set object detection model GroundingDINO (Liu et al., 2023) that generates rectangular box around the region/entity that needs be altered. After getting the box coordinates from GroundingDINO,, we further perform the im- age segmentation using the SOTA model SAM (Segment Anything Mask) (Kirillov et al., 2023), which takes point inputs for generating a segmenta- tion mask over the entity. These steps can be seen in Figure 4: ChatGPT outputs the entity to which the edit has be applied given the caption and the instruction (sky), DroundingDINO locates the entity (red rectangular box), and the SAM model further disentangles the sky from the mountains.
Ensuring Faithfulness To further account for faithfulness, we leverage techniques from Visual Question Answering (Antol et al., 2015) to ensure that the edited images are faithful to the instruction as well as the entities in the original image. We formulate relevant questions regarding the unmodi- fied elements in the image such as the presence specific objects or contextual information. We gen-
Generating Images using Stable Diffusion In- painting Stable Diffusion Inpainting is a latent

---

erate these questions using the Vicuna-13IB model (Chiang et al., 2023), a LLaMA model (Touvron et al., 2023) fine-tuned for instruction following. We use the edited caption available in the dataset to extract noun phrases and drop entities which are either locations or names of individuals. For example, in Figure 4, given the edited caption "But- termere Lake District with Aurora Borealis" and the extracted entities Buttermere, Lake District and Aurora Borealis, we drop Buttermere and pass the other two entities to the Vicuna model to generate question ceesstion-nsswer -nswer pairs. Figure 4 shows the gen- crated questions corresponding to the individual entities.
Dataset Type
Total Pairs
Instr-Pix2Pix (In-domain)
200
Magic- Magic-Brush Brush (Out-domain)
200
Visual-Metaphors (Out-domain)
65
Table 1: Test split and number of image-caption pairs.
For creating an out-of- test set, use two sources of examples. First, we consider the recently released MagicBrush dataset (Zhang al., 2023). This a manually annotated dataset for instruction instruction-guided image editing that covers multi- ple scenarios including single-turn and multi-turn edits. They sample real-world images from the MS- COCO (Lin et al., 2015) dataset, ask annotators to write instructions, and use the DALLE-2 (Ramesh et al., 2022) image editing platform to interactively synthesize target images. For our experiments, we only consider single-turn edits. We discard the Change Action instructions such as Make the per- son jump, Make the dog look away as they were not present in the original InstructPix2Pix data. We finally select 200 random images from the Mag- icBrush test set after the above considerations. Second, we use dataset of 100 DALLE- generated imperfect visual metaphor images paired with sspett-written natural language instructions improve them (Chakrabarty et al., 2023). We select 65 single-turn images with verbs that are already present in our in-domain test set.
For Remove or Delete instructions, we need to ensure that the edit entity not present in the re- sulting image. Along with the edited caption and the extracted entities, we also pass the edit instruc- tion to generate a question ensuring a successful edit. For instance, suppose the original caption is Buttermere Lake District with Aurora Borealis and the edit instruction Remove Aurora Borealis. We input the edited caption Buttermere Lake District, the entity lake district, and the edit instruction to generate the following questions: Is there a lake district in the picture? and Does the picture contain Aurora Borealis? along with the corresponding cor- rect answers Yes, No.
We input the questions and the image generated by Stable Diffusion Inpainting in the SOTA VQA- model BLIP-2 (Li et al., 2023) to extract whether entities are present in the edited image. We count the number of correct answers (No for remove in- structions, Yes for other instructions) for each of the three images generated using Stable Diffusion Inpainting and select the one having the largest number of correct answers.
3 Fine-tuning with Additional Supervision
Our training set curation pipeline led to a set of 52,208 high-quality instruction estruction-images pairs. We split this data in 10:10:10 for the training, valida- tion, and test sets, respectively.
We follow the same protocol as InstructPix2Pix training and use their codebase. For an image x. the diffusion process adds noise to the encoded la- tent producing a noisy latent E(x) where the noise level 2, increases over timesteps e T. We learn a network that predicts the noise added to the noisy latent 2 given image conditioning Gr and text instruction conditioning Cr. We minimize the latent diffusion objective and initialize the weights of our model with the InstructPix2Pix checkpoint. To support image conditioning, add input chan- nels to the first convolutional layer, concatenating žt and E(cr). ). All available weights of the diffusion model are initialized from the pre-trained check- points, and weights that operate on the newly added input channels are initialized to zero. We reuse the
2.3 Test Data creation
Our test set of 465 <image, instruction>> pairs is curated carefully across both in-domain and out-of- domain images to account for model robustness and generalization capabilities. To create an in-domain test set, we deployed filtering strategy where we only retained those examples in our test having CLIP-similaris score (Radford et al., 2021) less than 0.7 with every other training image. We fur- ther focus on multiple carefully chosen verbs such as Replace, Swap, Add, Turn, Change and generate 20 images per verb leading to 200 diverse <image, instruction"> pairs.

---

same text conditioning mechanism that was origi- nally intended for captions instead take as input the text edit instruction eT. We experiment with two different training strategies with additional su- pervision on top of the vanilla InstructPix2Pix.
variety of vision-language (VL) tasks. Given the high-quality referring segmentation results with X-Decoder, they further combine with an off-the- shelf Stable Diffusion image inpainting model and perform zero-shot referring image editing. 3
1. Bounding Box Supervision: During fine- tuning, the image conditioning e₁ is set to image with the bounding box around the region of interest (see Figure (a)).
Grounded Inpainting Grounded- -nnainnting module proposed by (Liu et al., 2023) deployed with gpt-3. 5..5-turo language model to identify the region/object of interest from the correspond- ing edit instruction. The pipeline then includes identifying the bounding box for the region of interest and generating binary segmentation mask before passing to the Stable Diffusion Inpainting model (Rombach et al., 2022b). We only provide the instruction to the inpainting module as opposed to the complete caption during the training dataset curation for fairesss to other baselines.
2. Segmentation Mask Supervision: During fine-tuning, the image conditioning сi set to the image with a denoised segmentation mask around the region/entity of interest. (see Figure 5 (b)).
InstructPix2Pix+ Bounx+inBonnBoxxx We use the model fine-tuned with a bounding box supervision signal described in section 3. During the infer- ence, the bounding box is constructed with the GroundingDINO model (Liu et al., 2023) using the entity extracted from the edit instruction by gpt-3.5-turbe
(b)
InstructPix2ii--nntin
We use the model fine-ture ine-ttned with a segmentation mask supervision signal described in section 3. During the inference, the segmentation mask is constructed with the Seg- ment Anything Mask (SAM) model (Kirillov et al.. 2023) using the entity extracted from the edit in- struction by gpt-3. 3.5-brbb..
Figure 5: (a) Bounding Box supervision (b) Segmenta- tion Mask supervision. Edit instruction: make the skirt red.
We initialized our model weights from the In- structPix2Pix model and finetuned for an additional 8k steps on NVIDIA-AI00GPUS We adopt the rest of the hyperparameters from the public Instruct- Pix2Pix public repository. Further details for hy- perparameters during inference are mentioned in Appendix A.
Evaluation
Automatic Evaluation using TIFA-Score We use a recent metric proposed by Hu et al. (2023), TIFA (Text-to-image Faithfulness evaluation with guestion-answeringg for evaluating the faithful- ness of the generated images to their text inputs. TIFA evaluates generated image using a two-stage pipeline: first, generates list of question- pairs that cover various aspects of the contextual information provided in the given caption, and then uses SOTA VQA models such as BLIP-2 to an- swer these questions and match the correct answers for faithfulness. This framework proposed for benchmarking diffusion models. The framework generates 7-10 question-answer pairs per image
4 Models
Below we describe how we use some the state- of-the-art baselines as well as ablations of our best model to generate outputs based on the test set instruction-image pairs.
InstructPix2Pix We use the off-the-shelf In- structPix2Pix (Brooks et al., 2023) model check- point.
Zou et al. (2023) released the X-Decoder model that provides a unified way to support all types of image segmentation and a
Dnstruct-X-Decode
'gi 'github. cowm/miccrooooft/e-eeeeee "tifa-benchmarr bi.a-bnnharrkggituu.iio

---

caption (modified, which covers the instructional edit aspects too). We average the score for individ- ual images over all these question pairs. The final score is reported as the average of the TIFA score over all the images in the test set.
the additional supervision signal lead better ed- its.
System
TIFA
Hacre I
Haooe O
Instruct-X -Deecderr
58.10
29.1
17.5
Grounded Inpainting InstrP2P
56.74
40.3
22.3
Human evaluation We randomly sample 100 examples from our test set with 50 images from in-domain and 50 images from out-of-domain. We recruit 31 annotators Amazon Mechanical Turk through rigorous qualification test. We require three distinct workers to do one HIT. Given an input image and an edit instruction, they are asked to judge if the output images from five systems satisfy the edit by choosing between Yes, Partially Yes, and No. Following prior work (Kayser et al., 2021; Majumder et al., 2021; Chakrabarty et al., 2022), map these to 1, 2 respectively and report the average Hggure. Additionally, we ask the annotators consider how faithful the output image is the input: the output image changes several elements that were beyond the scope of the edit instruction, they are asked to respond No. We require them to provide justification for their choice (see Figure 6) to prevent random guessing.
62.24
49.3
25.2
SattP2P-BOUNDINNE Inst:PPPEENTIT MASK
63.39
62.8
37.7
65.84
69.8
65.1
Table 2: Automatic evaluation using TIFA scores on the entire test set and human evaluation on in-domain(Hucor and out-of-domain (Hoore O) parts of the test set.
TIFA acts well as reference- free automatic eval- uation metric and useful in real-world settings gold images are not necessarily available. For in- stance, given the edited caption Red skirt balle- rina for the edit Make it red skirt (see top row in Figure 7), TIFA generates several <question, choices, answer> tuples such <Is there red skirt?, [Yes, No], Yes>, <Who is wearing the red skirt?, [Ballerina, Singer, Actress, Model], Bal- lerina>> <What color is the skirt?, [Red, Blue, Green, Ycllow], Red> which are then scored by a VQA model for correctness.
For human evaluation, as mentioned earlier, three MTurkers were recruited for each instance. The IAA using Krippendorff's (Krippendorff, 2011) is 0.58, indicating moderate agreement. Hu- man judges consistently preferred our Instruct- Pix2 Entity Mask model and the gap between our best model and the other baselines are sub- stantial. All judges unanimously agreed that the original InstructPix2Pix changes the images follow- ing the instructions; however, the resulting images exhibit excessive modification and lack photoreal- ism. For instance, for image (iv) in the top in Figure human judges the edit as bad and give the following justification: The entire dress has been changed to red and the background is a different color. These explanations shed light on user preferences corroborating the fact that humans like the edits to be precise and faithful. Qualitative examples in Figure 8 demonstrate that our model is capable of generalizing instructions and images (primarily illustrations) from completely different domain. See more examples in Appendix B.
Have the moon hot air baloon
Partially Yee
(three edit: adde afr hal born bot nos only changes the moon but rmplacee the entiiee landecape and coplaces the lakes and the rountains with green hills
(b)
Figure 6: Human evaluation framework (a) Input Image
with Edit Instruction (b) Example of bad edit
6 Results
Table 2 shows the performance of our model and the basclines the in-domain and out-of-domain parts of the test set in terms of both automatic and human evaluation. InstructPix2Pixx EntityMask appears to be the winning system with Instruct- Pix2Pix+ Boundii+oBunnBBBBB coming second. These re- sults indicate that our improved data combined with
Related Work
Prior work can be categorized primarily among text-guided image editing, involving the applica- tion of multiple models such as language mod-

---

Make
(h) Change the white couch to brown couch
Figure 7: Sample generations for an image from (a) instruct-pix2pix dataset (b) Magicbrush dataset. The images displayed following order (i) input-image, (ii) Grounded Inpainting,,i (iii) X-Decoder, (iv) InstructPix2Pix, (v)
(vi) InstrudPPixPPix-aatttyy
(a)
(b)
(c)
(d)
Figure 8: Sample edits showing <Input, InstructPix2Pixx InstructPix Mask> for the metaphors and corresponding edit instructions (a)l "eer song is like cloud of heavy smoke", "Add more smoke near the microphone" I] (b)( My heart a garden tired with autumn", "Change the heart such that is made of autumn leaves"] (c) ("My head is like vase growing", "Replace the body of the purple with the face man") (d)f river is ribbon wide". "Add landscape")
els and diffusion models in pipelined fashion and purely diffusion-basse generative models. Ap- proaches falling under the paradigm text-guided image editing can be further subdivided into global- description-guu -uudeed editing and local- deul-description guided editing. Imagic (Kawar et al., 2023) refines textual representation so that it aligns with a given image and then blends this with target description to create diverse image edits. On the other hand, Tex12LIVE (Bar-Tal al., 2022) employs a unique approach that teaches model to generate an edit layer. This layer is then merged with the input image, allowing for localized modifications. Coua-
iron et al. (2022) automatically generates mask highlighting regions of the input image that need be edited, by contrasting predictions of diffusion model conditioned on different text prompts. They further rely on latent inference to preserve content in those regions of interest and show excellent syn- ergies with mask-based diffusion.
Methods such as Prompi2Prompt Hertz et al. (2022b) utilize both local and global parts of the image using cross-attention network to perform edits. Approaches proposed by Avrahami et al. (2022b); Wang et al. (2023) work on regional de- scriptions for localized editing. Instruction guidedd

---

editing suggested by Brooks et al. (2023); Chen al. (2023); Fu al. (2020); El-Nouby et al. (2019) argues towards an approach to edit images via language instructions without explicitly men- tioning the contextual information. Recent works from Liu et al. (2023) explore strong object- detection model coupled with strong segmentation (Kirillov et al., 2023) model to edit via Stable Diffu- sion model (Rombach al., 2022a). Further work (Saharia al., 2022), (Richardson al., 2021), (Fu et al., 2020) explores mylistic-image editing including via generative models. Unlike several ex- isting prior works we focus on the faithfulness and specificity of object-centric edits. Like Zhang et al. (2023) we argue that high-quality training data and incorporating grounding is the key to achieving high-quality edits.
Limitations
While our best-performing model supports various edit types real images, we do not benchmark for global editing (e.g., style transfer). Our method can greatly enhance text-guided image editing, making accessible to more users without professional knowledge, boosting their efficiency. However, the risk of misuse for creating fake or harmful con- tent a concern. Therefore, implementing robust safeguards and responsible Al protocols is critical. Finally, while our data creation uses pipeline of best pest-performing state-of-the-an models, there is still potential for error in our training data. Addi- tionally, while TIFA score acts good reference- free metric for automatic evaluation it is relying on VQA model's correct answers which may be incorrect. Thus, we corroborate our results with human evaluation. Our study focuses on single- turn atomic instructions and does not show results on multi-turn instructional edits. Additionally, our method only works for natural language instruc- tions in English and does not handle instructions in other languages.
Recently there has been a growing interest in using text-to-image diffusion models for creative tasks such as creating illustrations or abstract art, Akula et al. (2023) release MetaCLUU, consist- ing of four interesting tasks (Classification, Under- standing, Localization, and Generation) related to metaphorical interpretation and generation of im- ages. Chakrabarty al. (2023) release dataset visual metaphors through collaboration between large language models and text-to-image models. These model-generated outputs while being impres- sive are often imperfect and require further edits. Our results on editing imperfect visual metaphors open up opportunities for content creators who can simply use natural language instructions steer Al-generated images to their liking.
Ethics Statement
The use of text-to-image generation models is sub- ject to concerns about intellectual property and copyright of the images generated since the models are trained on web-crawled images. We use the dataset which primarily con- sists of images from variety of mediums (pho- tographs, paintings, digital artwork). To ensure the collection of high-quality human annotations and fair treatment of our crowd workers, we have implemented a meticulous payment plan for the AMT task. We conduct a pilot study to estimate the average time required to complete a session. We pay our workers 18$/hr, which is above minimum wage. All data collected by human respondents were anonymized and only pertained to the data they were being shown. We do not report demo- graphic or geographic information so as to main- tain full anonymity. Workers were paid their wages in full immediately upon the completion of their work.
8 Conclusion
We address challenges in natural language-base based image editing tasks and provide novel approach enhance the quality of the training data. We im- prove the supervision signal and tackle the issues of underspecification, grounding, and faithfulness by leveraging advancements in segmentation, Chain- of-Thought prompting, and VQA. Our models fine- tuned on the improved dataset with enhanced su- pervision signal outperform the existing baselines on object-centric image editing both in terms of automatic and human evaluation. Moreover, our models showcase the capability edit faithfully on out-of-domain datasets. Overall our findings highlight the significance of high-quality annota- tion and grounded supervision signals for precise and faithful image editing.
References
Arjun R Akula, Brendan Driscoll, Pradyumna Narayana, Soravit Changpinyo, Zhiwei Jia, Suyash Damle, Garima Pruthi, Sugato Basu, Leonidas Guibas,

---

William T Freeman, et al. 2023. Metaclue: Towards comprehensive visual metaphors research. In Pro- ceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 23201-23211.
Alaacldin El-Nouby. Shikhar Sharma, Hannes Schulz, R Devon Hjelm, Layla EI Asri, Samira Ebrahimi Ka- hou, Yoshua Bengio, and Graham Taylor. 2019. Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction. In 2019 TEEE/CVF International Conference on Computer Vision (ICCV), pages 10303-10311.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433
Tsu-Jui Fu, Xin Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang. 2020. SSCR: Iterative language-based based image editing via self-supervised counterfactual reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4413-4422, On- line. Association for Computational Linguisties.
Omri Avrahami, Ohad Fried, and Dani Lischinski. 2022a. Blended latent diffusion. arXiv preprint arXiv:2206.02779.
Omri Avrahami, Dani Lischinski, and Ohad Fried. 2022b. Blended diffuusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 18208-18218.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aber- man, Yael Pritch, and Daniel Cohen-Or. 2022a. Prompt-to to- prompt image editing with cross attention control. arXiv preprint xrXiv:208..01626.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. 2022. Text2live: Text- driven layered image and video editing. Euro- pean Conference on Computer Vision, pages 707- 723. Springer.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aber- man, Yael Pritch, and Daniel Cohen-Or. 2022b. Prompt-to-pronee prompt image editing with cross attention control.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang. Mari Ostendorf, Ranjay Krishna, and Noah Smith. 2023. Tifa: Accurate and interpretable text-to- image faithfulness evaluation with question answer- ing, arXiv preprint arXiv.2303.11897.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023. Instructpix2pix: Learning to follow image edit- ing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 1892--18402.
Babjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Hui- wen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2023. Imagic: Text-based real image edit- ing with diffusion models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6007-6017
Tuhin Chakraharty, Arkadiy Saakyan, Dehanjan Ghosh, and Smaranda Muresan, 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7139-7159, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do Zeynep Akata, and Thomas Lukasiewicz. 2021. e-vil: A dataset and benchmark for natural language expla- nations in vision-language tasks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1244-1254.
Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang, Marianna Apid- ianaki, and Smaranda Muresan. 2023. I spy a metaphor: Large language models and diffusion models co-create visual metaphors, arXiv preprint 1.5.22345..422...
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. arXiv preprint 23X.::3304.02443..
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming- Wei Chang, and William W. Cohen. 2023. Subject-driven cext-to-image generation via apprenticeship learning.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Wei-Lin Chiang, Zhuohan Li, Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing g with 90%* chatgpt quality.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk. and Matthieu Cord. 2022. Diffedit: Diffusion-based semantic image editing with mask guidance. In The Eleventh International Conference on Learning Rep- rexentations.
Tsung Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dol- lár. 2015. Microsoft coco: Common objects in con- text.

---

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li. Hao Zhang, Jie Yang, Chunyuan Li. Jianwei Yang. Hang Su, Jun 7hu, al. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.088
Faisal Azhar, et 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 2302.117...
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J. Fleet, Radu Soricut, Ja- son Baldridge, Mohammad Norouzi, Peter Anderson, and William Chan. 2023. Imagen editor edit- bench: Advancing and evaluating text-guided image impainting.
Bodhisattwa Prasad Majumder, Mana-Maria Camburu, Thomas Lukasiewicz. and Julian McAuley. 2021. mmowledge-groundee self-rationnlizatiion via extrac- tive and natural language explanations. arXiv preprint arXiv: 22...100..777
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Nya Sutskever, and Mark Chen. 2021. Glide: To- wards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1..i:22122.10441
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint a 11903.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Su. 2023. Magicbrush: A manually annotated dataset for instruction-guided image editing.
Alec Radford, Jong Wook Kim, Chris Hallaey, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretehen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Xueyan Zou, Zi- Yi Dou. Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li. Xiyang Dai, Harkirat Behl, Jian- feng Wang, Lu Yuan, et al. 2023. Generalized decod- ing for pixel, image, and language. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 15I 15116-151
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot rext-to-image gen- eration. In International Conference on Machine Learning, pages 8821-8831 PMLR.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. 2021. Encoding in style: a stylegan encoder for image-to-image translation.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022a. High- resolution image synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser. and Björn Ommer. 2022b. High- resolution image synthesis with latent diffusion mod- cls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kam- yar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. 2022. Photo- realistic text-to-image diffusion models with deep language understanding. Advances Neural Infor- mation Processing Systems, 35:36479-360 36494.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro,

---

A Pormpts, hyperparameters, annotation
You are given an image description and the corresponding entities present in the caption. Generate a question per entity to check whether the image aligns with the text. Caption: handsome man wearing tuxedo and top hat in casual style clothes blurred mountain background. Entity: Handsome man, tuxedo top hat, casual style clothes, blurred mountain background Entity: Handsome man Question: Is the person in the picture handsome man?
A.1 Few-shot Prompts for filtering noise and handling under-speciffcation
We describe our few-shot prompts given to the ChatGPT model 1spt-3.5-turbo for our dataset- generation pipeline.
Answer: Yes
You will be given an input caption of an image and an instruction to transform by image editor. Sometimes, however, the instruction does not make sense the resulting transformation would result in nonsensical image. Based on the input caption and instruction, reason how the resulting image would look like and whether the resulting image would be possible to imagine. Provide your verdict whether the transformation is possible. If the verdiet is true, also state the entity. First, provide your reasoning, starting with the words "The resulting image would show... Then, return the verdict and the entity in JSON format.
Entity: Tuxedo and top hat
Question: Is the man wearing a tuxedo and top hat?
Answer: Yes
Entity Casual style clothes
Question: Is the man dressed in casual style clothes?
Answer: Yes
Entity: Blurred mountain background
Question: Is there blurred mountain background in the picture?
Answer: Yes
Caption:.....
Caption
Caption: Barn In Autumn Smoky Mountains by David Chasey
Table 4: Three shot prompt given Vieuna- generate question for ensuring faithfulness wrt original caption and instruction
Instruction: Change the barn to castle The resulting image would show a castle in mountains, which is a sensible image.
"verdict"; "true",
baselines and its ablations. For evaluation using TIFA, we use GPT-4 for generating questions and flan-t5-xxl version of the BLIP--2 model for visual question answering.
"entity": "barn")
Caption: Sligachan Bridge by English Landscupe Prints.
Instruction: change the bridge to a wooden ship, The resulting image would show a ship up in the air which does not make logical sense.
A.4 InstruetPix2Pix training data creation
[ ("verdict": "false".
"entity": "none")
Figure 9 shows the training data creation pipeline for InstructPix2Pix by (Brooks et al., 2023).
Caption:.....
Caption:.....
Caption:.....
A.5 Human annotation and explanations
Table 3: Five shot prompt given to ChatGPT (gpt-3.5 (ott3..--uurb) to elicit verdict on whether instruc- tion is valid in the context of input image caption and edit entity
Table 5 shows the rationale chosen by human judges for edits from different baselines. Image in row Entity Mask is deemed as perfect can be seen in the written explanation. The vanilla model edits the image too much beyond the scope as can be seen the images row 3 and of Table 5.
A.2 Few-shot Prompts for generating questions using Vicuna model
For question-answae pair generation to evaluate faithfulness during the dataset cutation, we provide three-shot prompt the Vicuna model provided in Table-4.
B Sample generations for the Metaphor and MagicBrush datasets
Figure 10 and 11 show sample image genera- tions from the Metaphor dataset with associated metappor-desccritionss and corresponding edit in- structions. Figure 12 and 13 show sample gen- erations for the Magicbrush dataset. The images displayed are following order (i) input-image, (ii) Grounded-Inpan Inpainting, (iii) X- X-Decoder, (iv) Instruct- Pix2Pix, (v) InstuutPix2Pix--Bnddngg.. (vi) InstruepPix2Pix--nniyyMa
A.3 Hyperparameters for training and inference
We finetuned the the InstructPix2Pix checkpoint from their official repository for Sk steps NVIDIA-A 100GPUs with learning rate of le 1. We take the rest of the hyper- param-tarseeters from the official implementation of InstructPix2Pix reposi- tory. For generation during inference, we set cfg- text=7.5 and cfg- -igge--ae=1.5 for instruct-pix2pl

---

(a) Generate text edits:
Instruction: "hove her ride dragon"
Input Caption: "photograph of gir riding horse"
GPT-3
Edited Caption: "photogroph of ridiing dragon"
(b) Generate paired images:
Input Caption: "photograph of girl riding horse"
Stable Diffusion Prompt2Prompt
Edited Caption: "photograph of riding dragon"
(<) Generated training examples:
"convert to brick"
"Color the cars pink"
"Moke it it by freworks"
"have her ride dragon"
Figure 9: Data creation pipeline for Instruct-Pix2Pix. (Brooks al., 2023)
Input
Instruction
Output
Explanation
This is good edit because the dress looks like something that could come out of medieval times armor)
Change the fashion medieval
This edit is partially good because it adds the street lamps but it removes the benches from the picture
Add street lamps
This edit makes the hair more spiky but also changes the face.
Make the hair of the boy spiky
put carrots on the styrofoam tray but took off the other items from the tray which wasn't 't a command
Put carrots on the styrofoam tray
Table 5: Images from Indomain test set( set(Top two) and Out Of Domain test (Bottom Two). The verdict chosen human judges for the four transformations are Yes ii) Partially Yes io iv) No

---

(a) The sun wounded deer", "Make the deer injured bleeding") 1
(b) A'A star crowns the Christmas Tree", "Add a christmas frees around the star"]
VIJT TANNTE
VITT TANNTE
VDT TANNTÉ
V
TU VTE
TANNTE
(6) ('Thought is vulture".
"Remove the text" "]
HEE PUL
POT
Her mouth mystical poison-ftower", "Change expression the girl portray anger"I
(e) My brain is like box of crayons", "Add crayons inside the bruin") "I
( ("Darey reputation for being butterfly gathering nectar". "Replace the man't head with butterfly"]
(g) (EEeeyyhhing like snowflake touching skin". "Change expression the boy be happy"1
Figure 10: Sample image generations from Metaphor dataset with associated metuphorr-dessriptinn and
corresponding edit instructions. The images displayed are following order (i) input-image, (ii) Grounded Grounded-Inpainting (iii) X-Decoder, (iv) InstructPix2Pix, (v) InstructPix 2 Bixx-Bouniingoxx (vi) Pnstructii22Pix-EEiiyyaaa

---

(a) He carried the knowledge or beliefs of tribe", "Add glowing lightbulbs near man 's head")
(b) l'He stands like perfect marbled statue, looking away from me" "Turn the into marble statue looking forward")
(c) | He was like block of cement", "Swap color of the head from yellow to grey"I
Figure 11: Sample image generations from the Metaphor dataset with associated metaphor-dessrripionn and
corresponding edit instructions. The images displayed are following order (i) input-image, (ii) Grounded Grounded-Inpaintingg (iii) X-Decoder, (iv) InstructPix2Pix, (v) InstructPix2Pix Pix-eiiiiii---BBungnBBoooo (vi) SestruutPixPPx++Eniiye

---

(a) "Remove bruccoli side"
(b) "Let the car turn blue."
cc) "let the remote turn into a VR headset."
(d) "Change the fire hydrant from to yellow." "
(e) "Make the teddy bear black."
"Can refrigerator blue?"
(g) "Have there be birthday cake the table."
Figure 12: Sample generations for images from the Magicbrush dataset. The images displayed are following order (i) input-image, (ii) Grounded-Inpainiinn (iii) X-Decoder, (iv) InstructPix2Pix. (v) InstmedPix2Pixx-ounddii (vi) PnstmmPPi22iix-EniiyMaa..

---

(a) "Let the bird turn yellow.
(b) "replace the lime green to red cup."
(c) "let there be painting instead sign."
Figure 13: Sample generations for image outputs from the Magicbrush dataset. The images displayed are
following order (i) imput-image, (ii) Grounded-Inpainn (iii) X-Decoder, (iv) InstructPix2Pix. (v) Instruct- Pix2Pix Pix2Pix+Bounding BoundingBox, (vi)


Reviews:
[{"review_id": "3hJpl6wi6a", "reviewer": "Reviewer_QHP1", "summary": "", "questions": "See above", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback\n\n**Reason to reject 1**: In object-centric edit we want the model to not change parts of the image that is not mentioned in the instruction. Our point is arguing for faithfulness and we think that faithfulness is compromised at the cost of creativity. When using GPT to determine if an instruction is feasible or not, 3 authors manually assessed more than 500 examples before going ahead and applying it on the entire data. There were errors in approximately 2% cases. In an ideal world every instruction needs to be verified by expert annotators but the data size is huge and we use ChatGPT to improve the InstrPix2Pix image which originally was automatically generated using GPT3 and was found to be very noisy. Finally, in the original InstrPix2Pix paper the edit instruction is generated on the original caption (see Fig 9 in Appendix) and so if the caption is underspecified and an instruction is nonsensical the issue is exacerbated. To ensure the quality of our training data, we opted to set aside samples with ambiguous captions. It is worth noting that there is no evidence to suggest that this approach omits high-quality, challenging editing image pairs from the initial InstrPix2Pix data. The authors of the original paper generated the data automatically, and didn't delve into the distribution of straightforward vs. challenging examples.\n\n**Reason to reject 2**: Yes there are artifacts in some images generated using box supervision, but not in all. A post processing step can easily get rid of these artifacts but since this was just an ablation we did not do anything further. Our best proposed model is InstrPix2Pix + Entity Mask which uses segmentation on top of the bounding box and produces the best result without any artifacts and the merit of our results should be based on that.\n\n**Reason to reject 3**:\nHuman scores are harsher because they only choose between [0,0.5,1] for YES,PARTIALLY YES,NO. TIFA score for an individual sample will be the percentage of <questions,answer>  pairs generated by GPT3 where the answer matches correctly with the response of the VQA model. For example if GPT3 generated 10 questions,answers given a caption and the VQA model gets 7/10 answers correctly on the edited image the TIFA score is 0.7 and would always be higher as it is an approximate measure of the degree of faithfulness.TIFA scores while helpful as a metric requires VQA models to work reasonably well, which can be a limitation as sometimes they can fail to answer certain questions (line 459-461) so human scores are more reliable in this aspect", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed data processing pipeline potentially increases the quality of existing datasets, which can be quite noisy.\n- The box and mask supervision demonstrates better performance than plain instruct-pix2pix.\n- I think the data cleaning pipeline and test set would be beneficial to the community, provided that the authors do the reproduction work well afterwards. ", "reasons_to_reject": "- I am not fully convinced by the motivation behind the cleaning steps. While I do admit that some editing instructions can be quite challenging to achieve, the underspecification also allows flexibility and creativity for editing models. In addition, using GPT to determine if an instruction is feasible or not seems a bit unreliable to me as the generated captions can also be underspecified. Therefore, if the caption fails to describe all components clearly, GPT may tend to judge the instruction as impossible and thus remove high-quality, hard-positive editing image pairs from the dataset. Maybe I have missed something, can the authors justify this point?\n- I think the box supervision is not a robust or plausible method given that Fig7 shows some generated artifacts, looking like an edge of a box or a red class label.  \n- The TIFA improvement is less significant than the human scores, which makes me wonder if the mask supervision only improves a small set of testing instructions given that they are trained on similar data samples or if it actually performs quite differently from instruct-pix2pix on many testing instructions but arrives at a similar TIFA score. Could the authors justify on the differences between TIFA and the human scores regarding the robustness of the former?", "questions_to_authors": "See above", "paper_topic_and_main_contributions": "The paper proposes a pipeline to improve the quality of language-guided image editing datasets. The authors also curate a test set consisting of both in-domain and out-of-domain instructions for benchmarking. They validate training mechanisms with box or mask supervision and show the improvement over previous editing models. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "24n7WqAQVO", "reviewer": "Reviewer_6zEB", "summary": "", "questions": "Please see Reasons To Reject", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback\n\n**Reason to reject 1**: Our novelty lies in identifying existing issues in one of the popular instructional edit papers, cleaning the noise, highlighting faithfulness issues and how to make it better. We will release the data, code and model, all of which are important contributions to the community. Finally, while our pipeline is a combination of existing modules it is in no way trivial and a successful execution leads to effective performance. Finally SAM+Inpaining [1] is published on April 13 on arxiv, is not peer reviewed and considered contemporaneous by ACL policy. In addition, the focus of our work is on following edit instructions, whereas SAM+Inpaininting requires explicitly specifying entities to be replaced and how to replace them.\n\n**Reason to reject 2:** Our training-data generation pipeline relies on the Grounding-DINO model to identify the entity and surround it with the bounding-box, which is then passed to SAM for generating a segmentation-mask. During this refining phase of training-data, we drop the instruction-image pairs if Grounding-DINO fails to detect the corresponding entity (not object-centric instructions) and for the cases where SAM partially (incorrectly) segments the entity, the image is discarded during the faithfulness (VQA) test. This 2-step check ensures that we curate the highest quality image-instruction pairs for our training data. During the inference phase, we carefully select object-centric instructions which ensure a bounding box around the entity from the Grounding-DINO module. SAM always generates a corresponding mask given a supervised set of points through background points/bounding boxes. SAM is highly efficient at segmenting the desired entity correctly and the performance of our pipeline is demonstrated through our evaluations on both in-domain and out-domain test-sets. \nHowever, in a human-AI collaborative environment, a user can always adjust the mask using additional foreground/background points as input to the SAM pipeline, to further enhance the edit quality.\n\n\n**Reason to reject 3:** Our paper focuses on object centric edits and faithfulness. Brighten the whole photo is a style transfer instruction which is already tackled by InstrPix2Pix. “Make it as a cartoon” can be handled by our pipeline. If “it” is a human or an object  that is mentioned in the caption  ChatGPT will detect the entity segment and the finetuned model will transform it to a cartoon.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ This paper is well-written and easy to follow.\n+ The proposed pipeline is elegant and training-free, which has the potential ability in adapting to various vision-and-language tasks.\n+ They demonstrate superior performance from both quantitative and qualitative results, outperforming the baseline methods.", "reasons_to_reject": "+ Despite being effective, the proposed pipeline seems to be a combination of existing modules. It is also similar to this concurrent work as SAM+Inpaining [1]. I am wondering if this achieves the EMNLP novelty bar.\n+ Since the pipeline relies on SAM, it will be better to discuss how well SAM performs in this image editing case. If SAM fails, how to address and make the editing forward?\n+ Some cases in image editing are not local manipulation (or not object-centric). For example, \"brighten the whole photo\" or \"make it as cartoon\". How to overcome this limitation in your pipeline?", "questions_to_authors": "Please see Reasons To Reject", "paper_topic_and_main_contributions": "This paper aims at a robust pipeline for instruction-based image editing. Starting from data preparation, they adopt external LLMs and chain-of-thought (CoT) reasoning to select feasible edit instructions as examples. Then, built upon segment anything model (SAM), they acquire the editing mask via visual grounding and perform image inpainting for the resulting image. This curated data can benefit the trained model and enhance the robustness of image editing.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] arXiv:2304.06790 Inpaint Anything: Segment Anything Meets Image Inpainting\n"}, "extra_text": {}}, {"review_id": "7DrzowRRVl", "reviewer": "Reviewer_1SU6", "summary": "", "questions": "1- The authors should clearly mention and highlight their contributions.\n2- Pls explain exactly how \"Segment Anything Mask\" is leveraged. What happens if the background needs to be fragmented and not consistent in a given image?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback\n\n**1- The resulting model is not compared against the SOTA models like Diffedit [1].**\nWe will definitely cite it. We tried to compare against more recently published models and the idea of masking and inpainting is done in our Grounded Inpainting baseline (https://huggingface.co/runwayml/stable-diffusion-inpainting). We found some community implementations of DiffEdit but not the actual code from the authors which makes it harder to claim a fair comparison with the authors’ model . Additionally their data was not in the form of natural language instructions and hence not entirely comparable. \n\n**2- The qualitative results, considering the fact that there is no GT for more reliable quantitative analysis, are not enough.**\nIn real world situations there is no ground truth so reference free quantitative evaluation is more reliable. We use TIFA scores for quantitative evaluation which is the state of art reference free metric accepted to ICCV 2023. TIFA correlates better than CLIPScore which is often used for text to image edit outputs. Finally,  for complex object centric edits, human evaluation is more trustworthy compared to any automatic evaluation. We performed human evaluation described in Section 5 lines 412-436.\n\n**Questions For The Authors:** \n\n**1- The authors should clearly mention and highlight their contributions.**\nOur contributions are explicitly stated in lines 123-139 in bullet points in introduction.  Our contributions are  \nImproving the quality of existing paired datasets used for image editing via natural language instructions with the help of recent advances in segmentation, chain-of-thought prompting, and visual question answering.\nCurating a test set of diverse non-noisy instructions, consisting of both in-domain and out-of-domain examples and conducting a thorough evaluation across SOTA baselines and our model ablations.\nDemonstrating that fine-tuning a diffusion model on our parallel data enhanced with supervision signal leads to a significant improvement over several compelling baselines in terms of faithfulness using TIFA scores as well as instruction satisfiability. Our human evaluation corroborates these findings.\n\n**Pls explain exactly how \"Segment Anything Mask\" is leveraged. What happens if the background needs to be fragmented and not consistent in a given image?**\nSegment Anything Mask is used as an additional supervision signal during training and inference. During training, we provide the InstructPix2Pix model with an input image with a superimposed mask filled with a salt-and-pepper noise as shown in Fig-5(b), the edit instruction and the corresponding after image (generated using our data-refining pipeline). By providing a mask that highlights the specific regions of interest, the model gains a clearer understanding of where to apply the edits, ensuring that the modifications align with the user's intent and achieving faithfulness by ensuring that only the specified regions are transformed, leaving the rest of the image intact. \n\nFor cases where the background needs to be fragmented such as instances with multiple objects of interest, the GroundingDINO module identifies multiple potential entities within the image, surpassing a predefined confidence threshold. Subsequently, SAM generates a segmentation mask that highlights all these predicted entities. These individual masks are merged to produce a final comprehensive mask thus accommodating instances that necessitate modifications over scattered elements in the image.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a procedure using the recent advances in computer vision and NLP domains to generate a reliable ground truth for text-guided image editing. Such a procedure can be used for generating more reliable supervised datasets in this field.", "reasons_to_reject": "1- The resulting model is not compared against the SOTA models like Diffedit [1].\n2- The qualitative results, considering the fact that there is no GT for more reliable quantitative analysis, are not enough.\n\n\n\n\n\n\n[1] Couairon, Guillaume, et al. \"DiffEdit: Diffusion-based semantic image editing with mask guidance.\" The Eleventh International Conference on Learning Representations. 2022.", "questions_to_authors": "1- The authors should clearly mention and highlight their contributions.\n2- Pls explain exactly how \"Segment Anything Mask\" is leveraged. What happens if the background needs to be fragmented and not consistent in a given image?", "paper_topic_and_main_contributions": "The paper proposes a method for improving the training data for text-guided image editing by benefitting from recent advances in segmentation, chain of thought prompting, and VQA. By conducting a subjective evaluation and leveraging the TIFA score it is shown that by fine-tuning the InstrPix2Pix model on the enhanced data, its performance improves.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Couairon, Guillaume, et al. \"Diffedit: Diffusion-based semantic image editing with mask guidance.\" arXiv preprint arXiv:2210.11427 (2022)."}, "extra_text": {}}, {"review_id": "m9aqOJlIBv", "reviewer": "Reviewer_Jm5D", "summary": "", "questions": "Please see my questions in Reasons to Accept and Reasons to Reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback\n\n**First, it is not clear how the paper is leveraging the supervision signals (bboxes and segmentation masks) in the finetuning. Section 3 is really weak and does not provide any details on this important step. For example, do you pass them as additional channels into the UNet? Please provide more details as it seems to be one of the important contributions of this work.**\n\n\nWe follow the same protocol as InstrPix2Pix training and use their codebase. For an image x, the diffusion process adds noise to the encoded latent $z = E(x)$ producing a noisy latent $z_{t}$ where the noise level increases over timesteps t ∈ T. We learn a network $\\theta$ that predicts the noise added to the noisy latent  $z_{t}$  given image conditioning $c_{I}$ and text instruction conditioning $c_{T}$\nWe minimize the latent diffusion objective and initialize the weights of  our model with a InstrPix2Pix checkpoint. To support image conditioning, we add additional input channels to the first convolutional layer, concatenating $z_{t}$  and $E(c_{I})$. All available weights of the diffusion model are initialized from the pre-trained checkpoints, and weights that operate on the newly added input channels are initialized to zero. We reuse the same text conditioning mechanism that was originally intended for captions to instead take as input the text edit instruction $c_{T}$\n\n\nWe first create the improved and filtered dataset as shown in Figure 4. We then fine-tune 2 separate models \n* InstructPix2Pix+ Bounding Box\n* InstructPix2Pix+Entity Mask\n\nThe training data for 1) has  image conditioning $c_{I}$ as an image with bounding box (Figure 5a) while training data for 2) has  image conditioning $c_{I}$  as an image with a segmentation mask on the desired entity/object. \nWe will add these details in the final paper if accepted, given the additional one page.\n\n**Next, in Figure 8, did you use the collected finetuned data for Instructpix2pix baseline? or Is the original data Instructpix2pix data used for both the baselines Instructpix2pix and InstructPix2Pix+Entity Mask. Please clarify.**\n\nThe original checkpoint provided by the InstructPix2Pix authors was used for the InstructPix2Pix baseline without any additional fine-tuning on our improved dataset. This checkpoint is finetuned with the newly refined image-instruction data pairs with two supervision signals as mentioned in the paper. For figure-8, the images used are a part of out-of-domain test-set from the Visual-Metaphor dataset (Chakrabarty et al) and we demonstrate the outputs from the InstructPix2Pix baseline and the InstructPix2Pix model fine-tuned with entity-mask.\n\n**It would be nice to add more discussion/insights comparing the prior work on T2I results with visual metaphors datasets such as MetaCLUE (see citation below).**\n\nWe will cite MetaCLUE. It should be noted that the METACLUE paper’s data is not publicly released which makes any discussion or insight difficult. Finally we are not trying to generate or classify Visual metaphors but rather editing imperfect visual metaphors. The data from Chakrabarty et al (2023) was used as they had natural language instructions paired with imperfect visual metaphors and their data was publicly available. \n\n**There are other presentation issues in the paper. For example, the captions of figures show labels such as (a), (b), etc. But for most figures I couldn’t find them labeled in the figure which makes it super difficult to understand the inputs and outputs of models.**\n\nWe will improve the presentation of the labels so that the input and output is more understandable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed pipeline of enhancing paired data with CoT prompting, Grounding DINO, SAM,  BLIP-2 is really interesting. Are the authors planning to release the collected dataset? Evaluation with TIFA scores and human evaluation is solid and nicely demonstrates the impact of the proposed dataset.", "reasons_to_reject": "I find the following major weaknesses of the work: First, it is not clear how the paper is leveraging the supervision signals (bboxes and segmentation masks) in the finetuning. Section 3 is really weak and does not provide any details on this important step. For example, do you pass them as additional channels into the UNet? Please provide more details as it seems to be one of the important contributions of this work. Next, in Figure 8, did you use the collected finetuned data for Instructpix2pix baseline? or Is the original data Instructpix2pix data used for both the baselines Instructpix2pix and InstructPix2Pix+Entity Mask. Please clarify. It would be nice to add more discussion/insights comparing the prior work on T2I results with visual metaphors datasets such as MetaCLUE (see citation below). There are other presentation issues in the paper. For example, the captions of figures show labels such as (a), (b), etc. But for most figures I couldn’t find them labeled in the figure which makes it super difficult to understand the inputs and outputs of models. \n\nAkula, Arjun R., Brendan Driscoll, Pradyumna Narayana, Soravit Changpinyo, Zhiwei Jia, Suyash Damle, Garima Pruthi et al. \"Metaclue: Towards comprehensive visual metaphors research.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23201-23211. 2023.", "questions_to_authors": "Please see my questions in Reasons to Accept and Reasons to Reject.", "paper_topic_and_main_contributions": "The paper addresses the challenges of underspecification, grounding and faithfulness in object-centric image editing with natural language instructions. The key idea of the paper is to enhance the quality of the paired data using recent advancements in segmentation, CoT prompting, and VQA. In addition, the work curates a test set consisting of both in-domain and out-of-domain examples. Furthermore, the paper proposes fine-tuning of the models with additional supervision signals such as bounding boxes and segmentation masks. Experiments demonstrate the utility of the proposed enhanced dataset and the fine-tuning strategies in improving automatic and human evaluation scores.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please see Reasons to Reject.", "Typos_Grammar_Style_and_Presentation_Improvements": "Please see Reasons to Reject."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a pipeline to enhance text-guided image editing by improving the quality of paired data through recent advancements in segmentation, chain-of-thoughts prompting, and VQA. While the proposed approach is intriguing and leverages important developments in computer vision and NLP, there are notable weaknesses in clarity and presentation. Specifically, the paper lacks detailed explanations regarding the use of supervision signals in fine-tuning and exhibits confusion regarding the treatment of fine-tuned data in experiments. Presentation issues, such as missing labels in figures, hinder the understanding of model inputs and outputs. Furthermore, the paper would benefit from a comparative analysis with other relevant datasets or methods, such as MetaCLUE. Despite these shortcomings, the use of TIFA scores and human evaluation adds credibility to the proposed dataset's impact, making it a valuable contribution to text-guided image editing with significant potential after addressing the mentioned issues.
Decision: Accept-Findings