Visual Instruction Inversion: Image Editing via Image Prompting
Authors: Thao Nguyen
Yuheng Li
Utkarsh Ojha
Yong Jae Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: image editing
diffusion models
visual prompting
TL;DR: We propose a framework for inverting visual prompts into editing instructions for text-to-image diffusion models.

Abstract: Text-conditioned image editing has emerged as a powerful tool for editing images.
However, in many situations, language can be ambiguous and ineffective in describing specific image edits.
When faced with such challenges, visual prompts can be a more informative and intuitive way to convey ideas.
We present a method for image editing via visual prompting.
Given pairs of example that represent the "before" and "after" images of an edit, our goal is to learn a text-based editing direction that can be used to perform the same edit on new images.
We leverage the rich, pretrained editing capabilities of text-to-image diffusion models by inverting visual prompts into editing instructions.
Our results show that with just one example pair, we can achieve competitive results compared to state-of-the-art text-conditioned image editing frameworks.

Visual Instruction Inversion: Image Editing via Visual Prompting
Thao Nguyen
Yuheng Li Utkarsh Ojha University of Madiisnn
Yong Jae Lee
https://thaoshibee hit.:/ibbbeittuuuub.oi//aaa.
Instruction
husly'
"i.pplt bull"
figer"
monbey
Text-conditionel
"Turm into painting"
Instruction
Test Image
InstruciPix2Pix
Visual Prompting
"..
pit
Before
After
Test Image
Ours
Figure 1: Image editing via visual prompting. Given a pair before-and-after images edit, our approach (bottom) can learn and apply that edit along with the user's text prompt to enable more accurate and intuitive image editing process compared to text-only conditioned approaches (top).
Abstract
Text-conditioned image editing has emerged as a powerful tool for editing images. However, in many situations, language be ambiguous and ineffective in de- scribing specific image edits. When faced with such challenges, visual prompts can be more informative and intuitive way to convey the desired edit. We present method for image editing via visual prompting. Given example pairs that represent the "before" and "after" images of an edit, approach learns text-based editing direction that can be used to perform the same edit on new images. We leverage the rich, pretrained editing capabilities of text-to-image diffusion models by invert- ing visual prompts into editing instructions. Our results show that even with just example pair,
achieve competitive results compared state-of-the-art text-conditioned image editing frameworks.
1
Introduction
In the past few years, diffusion models [37 38
40 have emerged as a powerful framework for image generation. In particular, text-to-image diffusion models can generate stunning images conditioned on a text prompt. Such models have also been developed for image editing [27 4 19 30
[54
13
16 39 26 i.e., transforming an image into another based text specification. As these
models rely on textual guidance, significant effort has been made in prompt engineering 49 12 48
which aims find well-designed prompts for text-to-image generation and editing.
But what if the desired edit difficult to describe in words? For example, describing your drawing style of your cat be challenging to put into sentence (Fig. 2 Or imagine that you want to transform a roadmap image into an aerial one could be difficult to know what the different colored
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

CT
"Make it a drawing"
"Turn into acrial photo"
Before
Text Prompt
Output
Before
After
Query
Output
Example
(a) Prior Work:
Text-conditioned image editing
(b) Our approach: "Visual prompting" image editing
Figure 2: Image editing with visual prompting, (a) Text-conditonned scheme (Prior work): Model takes an input image and text prompt to perform the desired edit. (h) Visual prompting scheme (Ours): Given pair of hefore-after images of an edit, we learn an implicit text-hased editing instruction, and apply it to new images.
regions in the roadmap image supposed to represent, leading to an incorrect output image. such cases, it would be easier, and more direct, to convey the edit visually by showing a before-and-afee example pair (Fig. 2p). In other words, language can be ambiguous when describing specific image edit transformation, while visual prompts can offer more intuitive and precise way to describe it.
Visual prompting for image editing has very recently been explored in 3 These works reformu- late the problem as an image in in-painting task, where an example image pair ("before" and "after") and query image are provided in single grid-like image. The target output is inpainted (by form- ing the analogy, bef ore: after = query: output). After training on a large dataset of computer vision tasks (e.g., edge detection, bounding box localization). these systems aim to perform any of those tasks during testing with in-context learning [3 46) without further fine-tuning, However, while they can work reasonably well for standard computer vision tasks such as segmentation and colorization, they cannot be used for general image editing tasks since large datasets for arbitrary edits are typically unavailable.
This paper investigates image editing via visual prompting using text-to-imen image diffusion models. Inspired by textual inversion [1O], which inverts visual identity specified by example image into the rich, pre-trained text embedding of a large vision-and-languagg model [38 32 for text- to-image generation, propose invert the visual edit transformation specified by the example before-and-afee after image pair into a text instruction. In particular, we leverage the textual instruction space that InstructPix2Pix [4] has learned. Since InstructPix2Pix directly builds upon pretrained Stable Diffusion model's vast text-to-image generation capabilities, while further finetuning it with 450,000 (text instruction, before image, after image) triplets, our hypothesis that its learned instruction space is rich enough to cover many image-to- image translations (i.e., image edits), and thus can be fruitful for visual prompt based image editing. Specifically, given a pair of images representing the "before" and "after" states of an editing task, we learn the edit direction in text space by optimizing for the textual instruction that converts the "before" image into the "after" image. Once learned, this edit direction can then be applied to new test image, together with a text prompt, facilitating precise image editing; see Figure
Our contributions and main findings are: (I) We introduce a new scheme for image editing via visual prompting. (2) We propose a framework for inverting visual prompts into editing instructions for text-to-image diffusion models. (3) By conducting in-depth analyses, we share valuable insights about image editing with diffusion models; e.g., concatenating instructions between learned and natural language yields hybrid editing instruction that is more precise: or reusing the same noise schedule in training for testing leads more balanced result between editing effects and faithfulness to the input image.
2 Related Work
Text-to-image Models. Early works on text-to-image synthesis based on GANs 53
57 21
were limited to small-scale and object-centric datascts, due to training difficulties of GANs. Auto-

---

regressive models pioneered the use of large-scale data for text- to-mm generation B3 34 52
However, they typically suffer from high computation costs and error accumulation. An emerging trend is large-scale text-to-image diffusion models, which are the current state-of-the-art in image synthesis, offering unprecedented image fidelity and language reasoning capabilities. Research efforts have focused on improving their image quality, controllability, and expanding the type of conditional 37 38
inputs [22] 9 2 [4
29 40 51]. In the realm of image editing, diffusion models are now at the forefront, providing rich editing capabilities through text descriptions and conditional inputs. In this work, we investigate how to use visual prompts to guide image edits with diffusion models.
Image Editing. In the beginning, image editing was primarily done within the image space. Previous GAN-based approaches utilized the meaningful latent space of GANs to perform editing [ [I
22 [T] The inversion technique [36 77 42 351 has been used to obtain the latent features of the input image, perform editing in the latent space, and revert the output to the image space. More recently, with the support of CLIP (32), a bridge between images and texts, image editing can now be guided by text prompts [31 I1] Recent models for text-conditioned image editing have leveraged CLIP embedding guidance and text-to-image diffusion models to achieve state-of-the-ar results for variety of edits. There are three main directions for research in this area: (I) zero-shot (exploiting the CLIP embedding directions, stochastic differential equations, or attention control) [K] 30 26 27 20];
(2) optimizing text prompts and/or diffusion models [19 39 T0 16 [6]; and (3) fine-tuning diffusion
models on supervised dataset [4] [54] In contrast prior works that rely text prompts to guide the editing, we aim to leverage visual prompts to better assist the process.
Prompt Tuning. Diffusion models have shown stirring results in text-to-image generation, but they can struggle to comprehend specific or novel concepts. Several works have focused on addressing this issue. Textual Inversion [10] learns specialized token for new objects, which can later be plugged in with natural language to generate novel scenes. Veresso [16) learns a specified text prompt for relation properties between two sets of images. Although continuous prompt can be more task-specific, a discrete prompt is typically easier to manipulate by users. PEZ 48) proposes method to discover prompts that can retrieve similar concepts of given input images. Instead of learning novel concepts for image generation, our work focuses on learning the transformation between example pair of images that is better suited for image editing.
Visual Prompting. Since proposed in NLP [5), prompting has been adapted by computer vision researchers. Unlike traditional methods that require separate models for each downstream task, visual prompting utilizes in-context learning to solve different tasks during inference. The first application of visual prompts was proposed by [3), where an example and query image are combined to form a grid-image. The task solver fills in the missing portion, which contains the answer. They showed that the task solvers can perform effectively on several tasks with only training on a dataset of Computer Vision figures. Later, 44 and [45] expanded the framework to increase the number of tasks that can be solved. Recently. Prompt Diffusion [47] introduced diffusion- based foundation for in-context learning. Although shows high-quality in-context generation, text prompt is still needed. Similar to textual prompts, not all visual prompts perform equally well. There are ongoing efforts to understand how to design a good example pair (55). The most similar idea to visual prompting is the seminal work of Image Analogies [14] and its variants [3 23). However, these works rely on pixel-to-pe pixel correspondences between example and test images, and struggle to transfer high-level idea edits (e.g.. "add dog" or "turn it into monkey"). Despite the success of visual prompting in solving wide range of standard computer vision tasks, the question of whether one can use visual prompting for image editing remains unanswered.
Framework
In this section, we present our approach for enabling image editing via visual prompting, First, we provide a brief background on text-conditioned image editing diffusion models (Section B.I). Section 3.2 and 3.3 describe how to invert visual prompts into text-based instructions. Finally, our full Visual Instruction Inversion algorithm is given in Section (.4]
Let (x. denote the before-and-after example of an edit. Our goal is to learn text-based edit er that captures the editing direction from x to y. Once learned, er can be applied to any new input image z', to obtain edited image y' that undergoes a similar transformation: + y x' + y/.
3

---

"As a puinting"
Cetip
Before
After
Testt
(optional)
InstruciPu2PPi
InstrudPPi2PP
Before
Output
mmse
Ey.
After
Instruction
Instruction Optimization
(b) Editing Direction
(c) Inference
Figure 3: Our framework. (a) Given an example before-amd iiforr--nddatie and-after image pair, we optimize the latent text instruction that converts the "before" image to the "after" image using frozen image editing diffusion model. (b) We leverage the CLIP embedding space help learn the editing direction. (c) Once learned, the instruction he applied to image achieve the edit. Optionally, the user also combine the learned instruction with a natural text prompt to create hybrid instruction.
To avoid confusion, use only one image pair example to describe approach. However, it is worth noting that our algorithm still holds for an arbitrary number example pairs.
3.1 Preliminaries
Diffusion models for image generation are trained on sequence of gradually noisier image x over series of timesteps i ., T. denoised variant of a noisy version of a at each timestep commonly denoted z; [15]. Initially, this approach was implemented in pixel-space (6), but it has now been extended to the latent space for faster inference and improved quality (37. Here, prior to the diffusion process, image is encoded by an encoder & to obtain the latent image 2z. which subsequently decoded by a decoder to convert the latent image back to the image space. The objective function is defined as follows:
The goal is learn denoising autoencoder cØ, which predicts
L = EE(x)). - cz(2.. l)
To enable diffusion models to take text prompts conditional inputs, 37 introduced domain- specific encoder To that projects text prompts to an intermediate representation Cr. This representation can then be inserted into the layers of the denoising network via cross rross-attention:
co c2(2,, t, cr)ll2
Conditioned text description cy, diffusion models synthesis stunning images. However, they are still not completely well-suited for image editing. Suppose that we want to edit image r to image y. conditioned text prompt cy- Text prompt er then needs align with our desired edit - and fully capture the visual aspects of x. which can be difficult. There are methods to help discover text prompts that can retrieve similar content 48 28
however, they typically cannot accurately describe all aspects of the input image x. To address this challenge, the idea adding the input image the denoising network was proposed in [4] 40) The input image can then be encoded as ç₁ = (zx) and concatenated to the latent image Zy. jointly guiding the editing process with text prompt cT. Based on this idea, InstructPix2Pix
fine-tunes the text-to-image diffusion model in supervised way to perform image editing. Its objective function is changed accordingly. as it now learns denoise noisy version y, which an edited image of a based on editing direction er:
C (rr.
ea ca t, (er,cr)ll
3.2 Learning to Reconstruct Images
Prior textual inversion methods [10 39 16] all utilize an image reconstruction loss. However, their aim is to learn to capture the essence of the concept in the image so that can be synthesized in new contexts, but not to faithfully follow pixel-level details of the input image which are required for image editing. The closest idea to ours is [19, 1, but it needs to fine-tune the diffusion model again for each edit and input image. We instead exploit pre-trained cext-conditioned image editing model, which offers editing capabilities, while avoiding additional one-tuning.
Given only two images {x. y) which represent the "before" and "after" images of an edit cr, the first and foremost objective to recover image y. We employ pretrained text-conditioned image editing model proposed in 4 where optimize the instruction er based the supervised pair {z, y). In

---

"Tum coke"
"harning condse
Tokenizer
Tokerizer
<mpp
<ped>
jod
Test Image
<PsP
<DS??
Instruction Optimzztion
(b) Instruction Concateration
"barning candie" "with "wilk orange Alrow
Figure Instruction details. Instruction Optimization: We only optimize part of the instruction
embedding er, called <ine>. (h) Hybrid instruction: During test time, we can add extra information into the learned instruction cy to further guide the edit.
our case, conditional image çf is the "before" image æ, and target image is the "after" image y. The objective function then adopted from Eq. Tas:
mmse
belg)..m..... (o or, 2r)
(2)
3.3 Learning Perform Image Editing
If we rely only the image reconstruction constraint (Eq. 2) we may learn description of the
edited image y, instead of the desired editing instruction. B1 30 has shown that the CLIP embedding [32] is a good indicator of the editing direction. 30 uses GPT-3 [5] to generate set of sentences for the "before" and "after" domains an edit; for example, cat ++ dog. The mean difference between the CLIP embeddings of these sentences represents the text editing direction "before" <> "after".
the difference between the CLIP embeddings of the "after" and "before" images to help learn the edit. Specifically, for an example pair {æ,, y}, we compute the image editing direction Ag-
In case,
4 = Exiip Eclip (z)
We encourage the learned instruction eT to be aligned with this editing direction (Figure 3p). To this end, we minimize the cosine distance between them in the CLIP embedding space:
Colip cosine cosine(Aa-yys CT)
3.4 Image Editing via Visual Prompting
Finally, given an example before-and-after image pair {x, y) we formulate the visual prompting as an instruction optimization using two constraints: Image reconstruction loss (Eq. 2 and CLIP loss (Eq. 3.. We provide an illustration of our framework training and testing in Figure Ba.c, and pseudocode in Algorithm| [ Our algorithm also holds for example pairs ((11,1)) y1),-... (Z Un)). In this case, Ar-ty becomes the mean difference all examples, and at each optimization step, we randomly sample pair {xt, y:)-
Once CT is learned, we can apply it to a new image Trest to edit it into Ylest. Moreover, our designed approach allows users to input extra information, enabling them combine the learned instruction er with an additional text prompt (FigureBe)) Te). To that end, we optimize a fixed number tokens of er only, which provides us with the flexibility concatenate additional information to the learned instruction during inference (Figure 4p)). This allows us to achieve more fine ame-grained control over the resulting images, and is the final default approach.
Evaluation
We compare our approach against both image-editing and visual prompting frameworks, on both synthetic and real images. Section 4.2 we present qualitative results, followed by quantitative comparison in Section 4.3 Both quantitative and qualitative results demonstrate that our approach not only achieves competitive performance to state-of-the-art models, but also has additional merits in specific cases. Additional qualitative results can be found in the Appendix.
5

---

Algorithm Visual Instruction Inversion (VISII)
1: Input: An example pair {æ. y}
2: Pretrained denoising model eg: Image encoder & CLIP encoder Edip
3: Number of optimization steps N; Number of timesteps T
4: Hyperparameters Actips Amsci Learning rate ?
5: I/ Start optimization
6: Initialize сT
Initialize instruction
7: Encode 2x E(x);
Zy = E(y)
Encode image D Compute editing direction
8:
Compute Ag- = Eciip - Ectip(2)
9:
for i 1, .N N do
10:
Samplet t L(,,,) T); N/ 1)
D Sample timestep and noise Prepare noisy version of 2y timestep t & Predict noise condition on æ Compute losses
11:
Zye add 2y timestep t
12:
= 60 1, CT,
13:
L= Amse | 2|12 + Adip (dipicosine(er Ax-ry)
14:
Update er = or VVL
end for
16: Output: CT
Example
painting Salvudor Dali"
watereolor painting"
"change dog"
"toonify his face"
Before
After
Test Image
Ours
SDEdit
InstructPix2Pix
Vis. Prompt
Figure 5: Qualitative comparisons. Our method learns edits from example pairs and thus can produce visually closer edited images to the target example than other state-of-the-ar baselines.
4.1 Experimental Settings
Training Setting. We use the frozen pretrained InstructPix2Pix to optimize the instruction er for N 1000 steps, 7 1000 timesteps. We use AdamW optimizer 25 with learning rate 0.001, Amse = 4, and detip 0.1. Text guidance and image guidance scores are set at their default value of 7.5 and 1.5, respectively. All experiments conducted 4 NVIDIA RTX 3090 machine.
Dataset. We randomly sampled images from the Clean-instructtix2Pix dataset 4 which consists of synthetic paired before-after images with corresponding descriptions. In addition, we download paired photos from [17 to test the models. Since some real images do not have edited versions, we utilize [54] with manual text prompts generate the after images with different edits.

---

Before
After
Example
Test Images
Before
After
Example
Test Images
Figure 6: A variety of edits can he performed for "Turn it into drawing/ painting" (Zoom in for details).
Evaluation Metrics. Following [4] we assess the effectiveness of our approach using the Directional CLIP similarity (1I) and Image CLIP similarity metries. However, as the CLIP directional metric does not reflect the transformation similarity between the before-after example and before-after output pair, we propose an additional metric called the Visual CLIP similarity. Specifically, we compute the cosine similarity between the before-after example pair and the before-after test pair as follows: cosine (Az- Ag
Svisual
We compare our approach to two main categories of baselines: Image Editing and Visual Prompting. For image editing, compare against InstructPix2Pix (4) and SDDEiit (26). which are the state-of-the-art. We directly use the ground-truth editing instruction for InstructPix2Pix and after descriptions for SDEdit. For real images, manually write instructions and descriptions for them, respectively. For visual prompting, we compare our approach against Visual Prompting [3] The Visual Prompting code is from the author's official repository, while SDEdit and InstructPix2Pix codes are from HuggingFace.
Baseline Models.
4.2 Qualitative Results
Figure 5 presents qualitative comparisons. As can seen, Visual Prompting K fails to perform the image editing task. Text-conditioned image editing frameworks, InstructPix2Pix 4 and SDEdit (26).

---

Ours
InstructPix2Pix
SDEdit
Number of images
50
40
20
20
20
0.4
0.6
0.8
1.0
-0.1
0.0 0.1
0.2 0.3 0.4
0.0
0.2
0.4
0.6
Image CLIP Similarity
Directional CLIP Similarity
Visual CLIP Similarity
Figure 7: Quantitative comparison, Histogram of Image, Directional, and Visual CLIP similarity scores. Our results comparable to state-of-the-arr text-conditioned image editing frameworks.
edit images based the provided text prompts, but fall short in producing edited images that are visually close to the "after" example. In contrast, our approach can learn the edit from the given example pair and apply it to test images. For example, in wolf dog (Fig. 5 row 3), only achieve successful domain translation from wolf to dog, but we also preserve the color of the dog's coat. Please refer the Appendix for more qualitative results.
Figure D demonstrates the advantage of visual prompting compared to text-based instruction. We can that one text instruction can be unclear describe specific edits. For example, the instruction "Turn it into a drawing" or "Make a painting" can have multiple interpretations in terms of style and genres. In these cases, by showing example pair, method can learn and replicate the distinctive characteristics of each specific art style.
4.3 Quantitative Results
We perform quantitative evaluation of our method against two baselines: InstruetPix2Pix 4 and SDEdit (26). Since Visual Prompting was not effective in performing image editing tasks, we did not include it in our comparison. We randomly sampled 300 editing directions, resulting in total of 1030 image pairs, from the Itean-InstructPix2Pix dataset.
We analyze the histograms of Image, Directional, and Visual CLIP Similarity (Figure 7 Results indicate that our method performs competitively the baselines. In terms of Directional CLIP Similarity, InstruetPix2Pix achieves the highest score, as it can make large changes the input image toward the editing instruction. Our method scores similarly to SDEdit, indicating that our approach can also perform well in learning the editing direction. Our approach is the most faithful to the input image, as reflected by the highest Image CLIP Similarity scores. Finally, for Visual CLIP Similarity, which measures the agreement between the changes in before-after example and before-after test images, our approach performs nearly identically to the two state-of-the-ee models.
5 Analysis
We next conduct an in-depth study better understand our method. For all of the studies below, we sample 100 editing directions (resulting in 400 before-and-after pairs in total) from the Clean- InstructPix2Pix [4] dataset. We show that both the CLIP loss and instruction initialization for achieving optimal performance. Additionally, we present some interesting findings regarding the effects of random noise, which can lead variations the output images.
critical
Losses. We ablate the effect of each loss in Table The additional CLIP loss helps improve scores in Visual and Directional CLIP Similarity [ТT], which reflect the editing directions. This shows that the CLIP loss encourages the learned instruction to be aligned with the target edit.
Initialization. Prior work utilizes coarse user text prompt (e.g., "sculpture" or "a sitting dog")) for textual initialization 10. which can be practical, but may not always effective. The reason is that natural text prompts can be misaligned with the model's preferred prompts [12]. We can also optimize upon a user's coarse input, however, we find that is more effective to initialize the instruction vector er to be somewhere close to the editing target; i.c., caption of the "after" image.

---

Table 1: Quantitative Analysis. We report Image, Directional, and Visual CLIP Similarity scores. Despite learning from only one example pair, our approach performs competitively to state-of-the- image editing models. ("Direct.": "Directional"; number training pairs; "Init": Initialization of
instruction; "GT" Ground-truth instruction; "Cap.": Image captioning of "after" image.)
Losses
Init.
Random noise
Fixed noise
MSE CLIP
GT
Cap.
Img f
Direct. f
Visual f
Img f
Direct.
Visual f
ground-truth no training
0.824
0.196
0.301
0.866
0.090
0.199
0.841
0.120
0.247
0.854
0.105
0.223
0.845
0.115
0.254
0.861
0.110
0.225
V
/
0.838
0.131
0.231
0.852
0.102
0.236
V
/
0.823
0.126
0.299
0.847
0.113
0.251
V
0.823
0,126
0.299
0.847
0.113
0.251
2
/
0.791
0.141
0.292
0.826
0.117
0,253
/
0.780
0.148
0.283
0.805
0.132
0.256
/
0.798
0.148
0.280
0.812
0.133
0.260
Example
Test
Fixed
Random
Figure 8: Fixed noise leads more balanced results. Different noises can lead large variations the output. Using the same training noises yields a balanced trade-off between editing manipulation and image reconstruction.
We evaluate both initialization strategies, including user input and captioning. To mimic a user's coarse input, we directly use ground-truth editing instructions from Clean- InstructPix2Pix dataset [4]. We employ [48] to generate captions for "after" images. Results are shown in Table As expected, directly using the caption as the instruction to InstructPix2Pix will not yield good results (Row 2), but initializing our model's learned instruction from the caption helps improve the Visual and Image CLIP Similarity scores. This indicates that the learned instruction is more faithful to the input test image that we want edit, while still retaining editing capabilities.
Noises. Text-conditioned models generate multiple variations of output images depending on the sampled noise sequence. This true for our approach too. However, for image editing, would prefer outputs that best reflect the edit provided in the before-and-aftor image pair, and preserve the test image as much as possible apart from that edit. We find that using identical noises from training in test time can help achieve this. Specifically, denote the noises sampled during the training optimization timesteps 1...7 as {cr.... ), which added to the latent image žy. We reuse the corresponding noises in the backward process during test time to denoise the output images. This technique helps to retain the input image content, as shown Table I However, there is trade-off between aggressively moving toward the edit, and retaining the conditional input. It is worth noting that in test time, we can also use random noises for denoising if desired. We visualize this phenomenon in Figure

---

(a) Fallure Case
What makes example?
Instruction: dinosaur"
After
Before
After
Example
Before
After
Learned Instruction
(b) Interesting Case
Test Images
Before
After
Test Image
Output
X
Figure 9: Discussion. (a) Failure Case: Our model can fail to capture fine details. (b) Interesting case: By preparing an image and segmentation pair, we can perform image segmentation. (c) Quality of example pair: One example does not work equally well for different test images.
Hybrid instruction. Finally, our approach allows users to incorporate additional information into the learned instruction. Specifically, we create a hybrid instruction by concatenating the learned instruction with the user's text prompt. This hybrid instruction better aligns with the given example pair while still following the user's direction. In Figure I transform "cat" - "watercolor cat". We demonstrate how concatenating extra information to the learned instruction (<ins>) enables both image editing (changing watercolor style) and domain translation (e.g.. "cat" "tiger"). The painting style is consistent with the before-and-after images, while the domain translation corresponds to the additional information provided by the Figure provides more qualitative examples. Applying InstruetPix2Pix 4 often does not yield satisfactory results, as the painting style differs from the reference image.
6 Discussion and Conclusion
We presented a novel framework for image editing via visual prompt inversion. With just one example representing the "before" and "after" of image editing task, our approach achieves competitive results to state-of-the-arr text-conditioned image editing models. However, there are still several limitations and open questions left for future research.
One major limitation is our reliance on a pre-trained model, InstructPix2Pixx As a result, restricts ability to perform editing in the full scope of diffusion models, and we might also inherit unwanted biases. Additionally, there are cases where our model fails, as shown Figure 4 where we fail to learn "add dinosaur" the input image, presumably because it is very small.
As we address the question of effectively using visual prompting with diffusion models, one might ask an interesting question in the reverse direction: Can diffusion models be used as task solver for downstream computer vision tasks? We find out that by preparing a foreground segmentation as green area, can learn instructions and apply them new images to obtain corresponding segmentations (Figure (p). However, further research needed to fully explore this question. We acknowledge that this question is beyond the scope of our current study, which is primarily focused on image editing. Additionally, visual in-context learning has been shown to sensitive to prompt selection [55] 3. Figure De shows cases where one example may not fit all test images. This shows that there are open questions regarding what makes a good example for image editing.
This work was supported in part by NSF CAREER IIS2150012, Adobe Data Science award, Sony Focused Research award, and Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. AS-2002-0011877 Development of Large Korean Language Model Technology for Efficient Pre- training).
Acknowledgements.
10

---

References
Alaluf, Y., Patashnik, O., Wu, 7., 7amir, A., Shechtman, E.,, Lischinski, D., Cohen-Or, D.: Third time's the charm? image and video editing with stylegan3. In: arXiv (2022)
[2] Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., Karras, T., Liu, M. ediff-i: Text-to-image -ii-tto-mmaee diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211 1122211011244 (2022)
[3] Bar, A.. Gandelsman, Y., Darrell, T., Globerson, A., Efros, A.A.: Visual prompting via image inpainting. arXiv preprint arXiv:2209.0000 (2022)
[4] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: arXiv (2023)
[5] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot Iearners. In: arxiv (2020)
[6] Dhariwal, P., Nichol, A.: Diffusion models beat gans image synthesis. In: Advances in Neural Informa- tion Processing Systems. vol. 34, pp. 8780-8794, Curran Associates, Inc. (2021), https::/ppooceddiee neurips http:///p...2aaaaae///////aaaaaa
www.tecccaaaaarpr.eeee pdf
[7] Dinh, T.M., Tran, A.T., Nguyen, R., Hua, B.S.: Hyperinverter: Improving stylegan inversion via hypernet- work. In: Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
Esser, P., Rombach, R., Ommer. B.: Taming transformers for high-resolution image synthesis. In: arXiv (2021)
[9] Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., Taigman, Y:: Make-a-scene: Scene-based text-to-image generation with human priors. In: arXiv (2022)
[10] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G.. Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. In: arXiv (2022). wttp://///...055//00012222220 01610, httpss.tttttss//ttttit wwtp:///.wrrraa/./....o.o
httgg/bb/220000 awwpaaba./2222//e 01618
[11] Gal, R., Patashnik, O., Maron, H., Chechik, G., Cohen-Or, D.: Stylegan-nada: Clip-guided domain adaptation of image generators. In: arXiv (2021)
[12] Hao, Y., Chi, Z., Dong, L.. Wei, F.: Optimizing prompts for text-to- generation. In: arXiv (2022)
[13] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompl image editing with cross atlention control. In: arXiv (2022)
[14] Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Pro- ceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. p. 327-340. SIGGRAPH '01, Association for Computing Machinery. New York, NY, USA (2001). https://di..or//1... 144453333399983222
httpp:://oi.. 2rg/10. 1145/383259 383295
[15] Ho. J.. Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: arXiv (2020)
[16] Huang, Z. Wu, Jiang, Y., Chan, K.C., Liu, Z.: ReVersion: Diffusion-baase relation inversion from images. arXiv preprint arXiv:2303. (2023)
[17] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. CVPR (2017)
[18] Karras, T., Laine, S., Aila, T: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
[19] Kawar, B., Zada, S., Lang, O., Tov, O., Chang. H., Dekel, T., Mosseri, L., Irani, M.: Imagic: Text-based real image editing with diffusion models. In: Conference Computer Vision and Pattern Recognition 2023 (2023)
11

---

[20] Kim. G., Kwon, T., Ye, J.C.: Diffusionclip: Text-guided diffusion models for robust image manipulation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 2426-2435 (June 2022)
[21] Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R.S., Torrahaa, A., Urtasun, R., Fidler, S.: Skip-thought vectors. arXiv preprint 16666:1506.06676 (2015)
[22] Li, Y., Liu, H. Wu, Q. Mu, F. Yang. J., Gao, J., Li, C., Lee, YJ..: Gligen: Open-set grounded text-to-image generation. In: 4xiv:2301.07099 (2023)
[23] Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual attribute transfer through deep image analogy (2017)
[24] Liu, Y., Gal, R., Bermano, A.H., Chen, B., Cohen-Or, D.: Self-conditioned generative adversarial networks for image editing, In: arXiv (2022)
[25] Loshchilov, I., Hutter, F. Decoupled weight decay regularization. In: arXiv (2019)
[26] Meng, C., Hc, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: Sdedit: Guided image synthesis and editing with stochastic differential equations. In: arXiv (2022)
[27] Mokady, R., Hertz, A., Aherman, K., Pritch, Y., Cohen-Or, D.: Null-text inversion for editing real images using guided diffusion models. In: arXiv (2022)
[28] Mokady, R., Hertz, A., Bermano, A.H.: Clipeap: Clip prefix for image captioning. In: arXiv (2021)
[29] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorcalistic image generation and editing with text-guided diffusion models. In: arXiv (2022)
[30] Parmar, G., Singh, K.K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-shot image-to-image translation. In: arXiv (2023)
[31] Patashnik, O., Wu, Z., Shechtman, E., Cohen-Oa Ooh D., Lischinski, D.: Styleclip: Text-driven manipulation of stylegan imagery. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 2085-2094 (October 2021)
[32] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I: Learning transferable visual models from natural language supervision. In: arXiv (2021)
[33] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hicrarchical textioociioil image generation with clip latents. In: arXiv (2022). http:///o..og////..5500/xxxx222.
htttps///rxxv..rrg
aba/2204 06125
[34] Ramesh, A.. Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. In: arXiv (2022)
[35] Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., Cohen Or, D.: Encoding in style: stylegan encoder for image-to- to-inage translation. In: IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) (June 2021)
[36] Roich, D.. Mokady, R., Bermano, A.H., Cohen-Or, D.: Pivotal tuning for latent-based editing of real images. In: arXiv (2021)
[37] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer. B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVP Conference on Computer Vision and Pattem Recognition (CVPR). pp. 10684-10695 (June 2022)
[38] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffiusion models. In: arXiv (2021)
[39] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-to- image diffusion models for subject-driven generation. In: arXiv (2022)
[40] Saharia, C., Chan, W., Chang, H., Lee, C.A., Ho, J., Salimans, T., Fleet, D.J., Norouzi, M.: Palette: Image-to-image difffusion models. In: arXiv (2022)
[41] Saharia, C., Chan, W., Saxena, S., Li, L., Whang. J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J.. Fleet, DJJ. Norouzi, M.: Photorealistic text-to-image diffusion models with deep language understanding, In: arXiv (2022)
12

---

[42] Tov, O., Alaluf, Y., Nitzan. Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for stylegan image manipulation. arXiv preprint arXiv:2102.02766 (2021)
[43] Subrtová, A., Lukáč, M., Čech, J., Futschik, D., Shechtman, E., Sýkora, D.: Diffusion image analogies. In: ACM SIGGRAPH 2023 Conference Proceedings. SIGGRAPH '23, Association for Computing Machinery, New York, NY, USA (2023). http:///ooi.rg///ieg 0t5p:/.......0.1114//54852253330 145/35884422.359155555
https://doi. org/10. 1145/
0119432.35915585
[44] Wang, X., Wang, W. Cao, Y., Shen, C., Huang. T.: Images speak in images: generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499 (2022)
[45] Wang. X., Zhang, X., Cao, Y. Wang. W., Shen, C., Huang, T. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 (2023)
[46] Wang. Z., Jiang, Y. Lu, Y., Shen, Y., He, P., Chen, W., Wang, 2., Zhou, M.: In-context learning unlacked for diffusion models. arXiv preprint arXiv:2305.01115 (2023), https://arxiv org/ www.bb/////. 01115
[47] Wang, 7., Jiang. Y., Lu, Y., Shen, Y. He, P. Chen, W. Wang, Z., Zhou, M.: In-context leamning unlocked for diffusion models. arXiv preprint arXiv:2305.01115 (2023), httpa: //aa:i/.//xiv..//// wwt./ogg///////ee.ooooooo 2200. 01050
[48] Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, Goldstein, T:: Hard prompts made easy: Gradient- based discrete optimization for prompt tuning and discovery. In: arXiv (2023)
[49] Witteveen, S., Andrews, M.: Investigating prompt engincering in diffusion models. In: arXiv (2022)
[50] Xu, T., Zhang, P., Huang, Q. Han Zhang, Z.G., Huang, X., Hc, X.: Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In: CVPR (2018)
[51] Yang, Z., Wang, Gan, Z., L., Lin, K., Wu, Duan, N., Liu, Z., Liu, C., Zeng, M., Wang, L.: Reco: Region-controlied text-to-image generation. In: arXiv (2022)
[52] Yu. J., Xu, Y., Koh, J.Y., Luong. T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B.K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang. H., Baldridge, J., Wu, Y.: Scaling autoregressive text-to-image generation. In: arXiv (2022)
models for content-rice
[53] Zhang, H., Xu, T., Li. H., Zhang, S., Wang, X., Huang. X., Metaxas, D.: Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: arXiv (2017)
[54] Zhang, L., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: arXiv (2023)
[55] Zhang, Y., Zhou, K., Liu, Z. What makes good examples for visual in-context learning? In: arXiv (2023)
[56] Zhang, Z. Han, L.. Ghosh, A., Metaxas. D., Ren, J.: Sine: Single image editing with text-to-image diffusion models. arXiv preprint 11X0v:2212.04489 (2022)
[57] Zhu, M., Pan, P., Chen, W. Yang, Y: Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In: arXiv (2019)
Appendix
A Textual Inversion vs. Visual Instruction Inversion
Textual Inversion [0 39 is a method to invert a visual concept into corresponding representation in the language space. In partieular, given (i) a text-to-image pre-trained model and (ii) some images describing visual concept (e.g., particular kind of toy; Figure I0 bottom row), Textual Inversion learns new "words" in the embedding space of the text-to-image model represent those visual concepts. Once these "words" are learned for that concept, they can be plugged into arbitrary textual descriptions, just like other English words, which can then be used to create the target visual concept in different contexts. Instead of learning the representation for an isolated visual concept, our approach (Visual Instruction Inversion), learns the transformation from a before-and-after image pair. This learned transformation is then applied to a test image to achieve similar edit "before" - "after".
13

---

"Painting <eat-toy the style Monet"
Example
"Painting of ceat-tors the style cwwtercooo-pootraiiss
<ino
<ing
Before
After
Test Image
Ours
coul-doyck
centereukorqurtziiss
<ins
(a) Textual Inversion
(b) Visual Instruction Inversion
Figure 10: Ours vs. Textual Inversion. (a) Textual Inversion inverts a visual concept object (e.g., particular <cat-toy>) into word embedding. This optimized word embedding can then be combined with textual description generate novel scenes. (h) Our Visual Instruction Inversion learns the transformation <ins>: "before" + "after" a given hefore-and-after image pair. This learned instruction can then applied to new test images to perform the same edit.
Applicability of Textual Inversion for image editing. Given these differences with our proposed method, we now try to see if Textual Inversion can be used for image editing. Textual Inversion can generate a "painting of <cat-toy> in the style of Monet" by using the learned word <cat-toyy from example images (Figure Top, Row 1). However, the synthesized images often only capture the essence of the objects, and disregard the details of the input images. As result, textual inversion is suitable for novel scene composition, but is not effective for image editing.
On the other hand, our Visual Instruction Inversion does not learn novel token representations for objects or concepts. Instead, we learn the edit instruction from before-and-after pairs, which can be applied to any test image to obtain corresponding edits. This allows us to achieve fine-grained control over the resulting images. For example, by providing photo <cat-toy>, one before and one in specific impressionist style, learn the transformation from before impressionist, denoted as <ins>, Once learned, this instruction can be applied to new <cat-toy> images to achieve the same impressionist painting style, without losing the fine details of the test image (Figure TOp, Row 1).
One might suggest an alternative approach to image editing using Textual Inversion, which involves learning two tokens: one for the object and another for the style (e.g., "Painting of <cat-toy in the style of <watercolor pawllor-porraats>>"
Figure (Row 2) shows the results this approach. As can be seen, Textual Inversion still often introduces significant changes that deviate from the original input image. Thus, Textual Inversion not suitable for accurate image editing.
B Additional Qualitative Comparisons
We present qualitative comparisons with other Hate-of-thhe-art text-conditioned image editing methods, Imagic [19] and Null-text Inversion [27 (Figure II]. These methods can generate outputs based on given text prompts, such as "A watercolor painting of cat" (Row 3). However, the outputs often do not match the given reference. The text prompts can also ambiguous and result in unsatisfactory outputs, illustrated by the case of "A character in Pixar movie" (Row 1).
Another challenge is the inconsistency of lext-conditioned models. where the same text prompt can produce different outputs for different test images. For example, the text prompt "A frozen waterfall" (Row 6) generates different water colors (blue vs. white) when applied different test images (Before-and-afiee pair is from (27)). Our method is more consistent in this case, the learned instruction might have learned the water color.
C Implementation Details
We use the pretrained instruction initialization 48), we set the caption length for after image to 10 tokens. However, this specific caption length does not affect the optimization algorithm. We can optimize initialization
as the CLIP Encoder in our approach. For
14

---

Example
"A character a Pixar movie" "
"A with marionette-like Jace"
"A watercolor painting of cat"
"A from western"
"A vase of tulips"
"A frozen waterfall"
"A chocolate cake"
Before
After
Test
Ours
== 0.6
Imagic
20066
Null-text
Figure 11: Additional qualitative comparisons Imagic 19 and Null-text Inversion 27
Imagic and
Null-text Inversion
match the reference image as they perform edits based on ambiguous text prompts (Row 1-4); or exhibit inconsistency in producing outputs for the same prompt across test images (Row 6). In contrast, our method produces visually closer edited images to the before-after pair while demonstrating improved consistency by using the learned instructions.
instructions of varying lengths (up 77 tokens). It takes roughly minutes to optimize for one edit, and seconds to apply the learned instruction to new images.
Specifically, during the optimization process, we freeze the tokens representing the start of text (<l startoofextt)) end of text (< eedoffeext |>), and all padding tokens after end of text (< endoftext()), We only update the tokens inside the text prompt, called <ins> (between startoffext/ and </endoftext| (Figure
Photo Attribution
Elsa (Human): reddit.com///Frozen
Disney characters: princess.disney.com
Toy Story characters: toystory.disney.com
Toonify faces: poonify.photos
15

---

Text cett-conditonee
Instruction
butterfley
impresaionist puiniing"
Instruction
Tesi Image
InstruciPix2Px
Tra with futteefilee
Visual Prompting
<iSs>
origuni"
Help
mariguidd
Mily
Before
After
Test
Ours
"enufesule gon"
fenull
Jemali with morey"
Text Text-conditione
Instruction
fomale"
Arion"
samlowerr"
"Thrn lato stiff from a western"
Instruction
Test Image
InstruciPa/PPix
"on fenule
fommlle
"... fomale with ruses
Visual Prompting
</1?>
Jemale"
Asion"
with o gue"
with sunflowers"
Before
After
Test Image
Ours
Figure 12: Hybrid instruction. We concatenate extra information into the learned instruction cy to navigate the edit. (Zoom for details.)
Girl with Pearl Earring: wwkkpppiaggfi--ithhhpppeae-aaarr
Mona Lisa: wikipediaamonna-issi
The Princesse de Broglie: wilkppedaa/rinnessedd---rriii
Self-portrait in a Straw Hat: wikkeediiaselffporrrriiiiasaaaaa
Bo the Shiba and Mam Cat: instagran/avoshiibe
<cat-toy> and wwwpercooooo--oorrrtteeee concept: boogrigggae.coossdcoonees----//p
Gnochi cat, waterfall, and cake images from Imagic [19] and Null-text Inversion [27]
16


Reviews:
[{"review_id": "HLLGjHM88e", "reviewer": "Reviewer_uc51", "summary": "In this paper, the authors propose a new method that can perform visual prompting via a pair of exemplar images through a pretrained text-based instruction image editing model. The method introduced in this paper only requires optimizing over the text conditioning vector in order to perform visual instruction editing. The authors have shown strong qualitative results demonstrating various application scenarios.", "questions": "1. How similar does the exemplar visual prompt and query image should look? Can the authors give some additional ablation study on the similarity between the visual prompt and the query images?\n2. How long does it take to sample one image?\n3. Since CLIP is not a perfect encoder and has various known limitations, I am wondering how well this method handles tasks such as color changes or duplicating objects.\n4. Related to Weakness (1), I think it is also possible to report more standardized metrics such as KID (fidelity score for small number of samples), IoU and (masked) LPIPS with tasks like semantic segmentation map to image and image compositing. Can the authors report some standardized metrics for some of these tasks?\n5. Why do more examples hurt the performance (according to Table 1)?\n\n~~I am happy to raise my score if the authors can add experiments to address my concerns.~~\n\n***After the rebuttal discussion, I would like to raise my score from 5 to 6.***", "limitations": "Although the imitations are extensively discussed in the paper, the discussion of potential ethical issues is missing. Since this paper uses a pretrained large image generative model, which is known to have various societal issues, I would highly recommend the authors to include a broader impact statement in their paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This model introduced in this paper solves a very interesting and important application problem that is exemplar-based visual prompting of image generative models. The paper is very well written and the results look very promising and this model can be easily directly applied in a large range of intuitive applications by daily users. It also leverages a pretrained model and only requires optimizing a vector, which makes it more appealing in applications. The authors have also performed thorough ablation study.", "weaknesses": "1. The quantitative evaluations are not very comprehensive. The authors only sampled 1k images to perform quantitative analysis, and did not report any fidelity scores. For editing controllability, they only report CLIP related scores, which is not very representative in many circumstances [1].\n2. When explaining the method, the authors sometimes mix the prior work with their contributions. A prime example would be Section 3.2, which is redundant because it is the same as InstructPix2Pix, but sectioning in this way makes it look like a new contribution at the first glance. I would recommend the authors to make clear distinctions between their contributions and the prior literature.\n\n[1] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, Candace Ross. “Winoground: Probing vision and language models for visio-linguistic compositionality”. CVPR 2022. https://arxiv.org/pdf/2204.03162.pdf\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback! We address your questions/concerns below.\n\n**Only sampled 1k imgs to perform quantitative analysis**\n\nWe believe that ~1000 images are sufficient to validate our approach. This is in line with other related work; e.g., Imagic (CVPR 2023): ~100 image pairs; Null-text Inversion (CVPR 2023): ~1000 test images.\n\n**[...] did not report any fidelity scores**\n\nThanks for your suggestion! Below, we present new experiment using the LPIPS score [1] for 25 in-the-wild image pairs.\nTo recap, we use a **before-and-after** image pair to learn an edit, then apply it to the **test** image to get an **edited** version.\n\nWe use LPIPS using two ways:\n- LPIPS(test, edited): Measures the similarity between the test and edited image. Lower score indicates that the edited image has similar image fidelity to the test image.\n- LPIPS(after, edited): Assesses how the edited image aligns with the after image. Lower score indicates that the edits successfully follow the visual cues provided by the after image.\n\n||SDEdit [3]|Ip2p [4]|Ours|  \n|---|---|---|---|  \n|LPIPS (test, edited)|**31.34**|46.40| 43.73|  \n|LPIPS (after, edited)|71.42|71.15|**62.52**|\n\nResults show that our model demonstrates competitive performance to state-of-the-art baselines. However, it is worth to note that while an edit may lead to significant changes, achieving lower LPIPS(test, edited) scores could indicate less meaningful edits, as the edited image closely resembles the original test image.\n\nFurthermore, in LPIPS(after, edited) score, our model scores lowest. This suggests that our edited images align more perceptually with after images compared to text-conditioned methods. Note that the baselines does not have access to the before-and-after pair – However, this precisely supports the key idea of our approach; i.e., using before-and-after images enhances intended edits which may not sufficiently be captured with text alone.\n\nIt is worth highlighting that real dataset for visual prompting-based image editing is lacking. Substantial ongoing research efforts are dedicated to creating diverse and precise paired datasets [2]. As a result, we were unable to compute on a larger dataset.\n\n**Standardized metrics for tasks like semantic segmentation**\n\nAs we focus on image editing, we did not report any downstream computer vision tasks (i.e., semantic segmentation, image compositing), which are beyond our scope.\n\n**How similar does the exemplar visual prompt and query image should look?**\n\nAs long as the example reference images and the query image are roughly in the same domain, there are no constraints on how similar they should be.\nFor example, In Fig. 3 (Last row), different people are in example and test images. In Fig. 6 (First row), backgrounds and people in example and test images fully change.\nOn the other hand, for entirely different domains (e.g., dog->cat as example, landscape as query), our method will not work.\n\n**How long does it take to sample one image?**\n\nIt takes 4 seconds to sample one image.\n\n**Since CLIP is not a perfect encoder [...] how well this method handles color changes or duplicating objects.**\n\nQualitative results about color changing cases can be found in our main paper:\n- Fig. 3 (Row 3: gray wolf -> brown dog)\n- Fig. 6 (Row 2: green grass field -> brown grass field)\n\nIt can also be spotted several times in our Supplementary (index.html), e.g., ID: 0207894 (red -> blue hair)\n\nFor \"duplicating objects\" it seems like your question is: What happens when there are two visually similar objects in a test photo? Please refer to ID: 0307987 - boats -> camels (Supplementary, index.html) as an example. The results show that each boat shown in the test images will be transformed into a camel. For more results, please see ID 0258921 (cow -> giraffe), ID: 0016482 (human -> cheetah).\n\nWith all the cases we have listed above, we see that our approach works reasonably well with a CLIP encoder.\n\n**Why more examples hurt the performance (Tab. 1)?**\n\nMore examples improves the performance. Recall that in diffusion-based models, different noises lead to different outputs (Fig. 6). Thus, in Tab. 1 (Column 6-8, \"Random noises\"), quantitative results can vary a lot.\n\nHowever, we find that using the same noise sequence from training to test leads to more stable results (Sec. 5 - Noises). Based on this observation, we have more reliable quantitative results in Tab. 1 (Column 9-11, \"Fixed noise\"). In particular, more example pairs (1 -> 4) increase Directional CLIP score (0.113 -> 0.133).\n\n**[...] the authors sometimes mix the prior work with their contributions. [...] Sec. 3.2 is redundant because it is the same as InstructPix2Pix [...]**\n\nThanks for your feedback. We do not want to cause any confusion here.\nWhile we use the same image reconstruction loss (MSE loss) as in InstructPix2Pix [4], our goals differ.\n-   InstructPix2Pix finetunes the model: update $\\epsilon_{\\theta}$, fixed $c_T$\n-   Ours optimizes the instruction: fixed $\\epsilon_{\\theta}$, update $c_T$ (As presented in Algorithm 1, Line 14)\n\nWe will change Line 141:\n“We  ̶f̶o̶l̶l̶o̶w̶ ̶t̶h̶e̶ ̶s̶a̶m̶e̶ ̶s̶t̶r̶a̶t̶e̶g̶y̶ ̶a̶s̶ ̶[̶4̶]̶ employ a pretrained text-conditioned image editing model proposed in [4] [...]”\n\n**Although the limitations are extensively discussed [...] discussion of potential ethical issues is missing.**\n\nIt is true that our method might inherit unwanted bias from diffusion models (As we briefly mentioned in Sec. 6, Line 256). We will further clarify potential ethical issues in our revision.\n\n*Reference:*\n\n[1] Richard et al., *The Unreasonable Effectiveness of Deep Features as a Perceptual Metric*, CVPR 2018\n\n[2] Kai et al., *MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing*, arXiv 2023\n\n[3] Chenlin et al., *SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations*, ICLR 2022\n\n[4] Tim et al., *InstructPix2Pix Learning to Follow Image Editing Instructions*, CVPR 2023"}, {"review_id": "i3QuXRarco", "reviewer": "Reviewer_h5yQ", "summary": "This paper investigates image editing via visual prompting useful when textual descriptions cannot describe desired edits. The proposed framework inverts visual prompts into editing instructions and learns directions in the text space of the pretrained instruct pix-to-pix model. This edit direction is learned from a pair of query and target images to generate desired image editing. Results suggest, one example is sufficient to learn such directions.", "questions": "Is it possible to enhance the quality of editing directions by providing more examples (before and after images)? Also, what is the average time required to find a single editing direction?", "limitations": "The paper needs to properly acknowledge previous work on image analogies and lacks originality in its approach. Other related works should be mentioned and compared for a comprehensive analysis, which is currently lacking.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- It is fascinating to see how visual cues can be translated into text-based editing directions, especially when the edits are difficult to convey through written instructions alone. \n- The editing approach that is learned can be used to modify new test images with impressive accuracy. It is impressive that comparable results can be achieved even with just one example pair. \n- Practical insights into image editing with diffusion models, including the potential to apply the same noise schedule for both training and testing is also interesting ", "weaknesses": "The paper has a significant weakness in that it fails to acknowledge seminal work on image analogies, specifically, the research conducted by Hertzmann et al. (2001) and their deep learning adaptation, deep image analogies (Liao et al., 2017). While the proposed approach shares similarities with the concept of image analogies, the use of instruct pix to pix with a similar analogy lacks novelty, except for the editing directions found in textual embedding space. \n\nFurthermore, the paper neglects to mention other related works such as SINE: SINgle Image Editing with Text-to-Image Diffusion Models (CVPR 2023) which employs style transfer for image editing through a diffusion model as seen in Figure 11 and Section 4.4 of their paper similar to image prompting. Additionally, MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation (AAAI 2023) is also absent in the current work. It is crucial to compare and discuss these related works for a comprehensive analysis, which is currently missing in the manuscript.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our paper \"fascinating\" and \"practical\". We answer your questions/concerns as below:\n\n**The paper has a significant weakness in that it fails to acknowledge seminal work on image analogies, specifically, the research conducted by Hertzmann et al. (2001) and their deep learning adaptation, deep image analogies (Liao et al., 2017)... Other related works should be mentioned and compared for a comprehensive analysis, which is currently lacking.**\n\nThank you for your suggestion.\nWe agree that *the seminal image analogies paper should have been cited*.\nWe will include Image Analogies [1], Deep Image Analogies [2], and other related works in the Related Works section on Visual Prompting (Line 87-99).\n\nHowever, *we do not think that we lack comprehensive analysis*.\n\nWe identify our work as being in the intersection of Visual Prompting (1) and Image Editing (2).\nThus, we conducted extensive comparisons against state-of-the-art frameworks in these two domains:\n\n- (1) Visual Prompting: Our work is inspired by Visual Prompting via Image Inpainting (NeurIPS 2022) [3].\nTo some extent, we believe that this work can be viewed as a modern deep learning variant of Image Analogies.\nSo comparing to Visual Prompting can be represented as comparing to one of the most recent state-of-the-art work in Image Analogies.\n\n- (2) Image Editing: We conducted quantitative and qualitative experiments on a handful of state-of-the-art baselines in image editing (e.g. InstructPix2Pix [4], SDEdit [5], Imagic [6], Null-text Inversion [7]).\n\nFurthermore, we also presented comparisons between ours and Textual Inversion method [8], which can be found in the Supplementary material (Section A).\n\n**While the proposed approach shares similarities with the concept of image analogies, the use of instruct pix to pix with a similar analogy lacks novelty, except for the editing directions found in textual embedding space. The paper lacks originality in its approach.**\n\nWe respectfully disagree (and we believe that other reviewers disagree too; e.g. [#VA7A](https://openreview.net/forum?id=l9BsCh8ikK&noteId=dVSnD0IySj_): \"proposed method [...] is novel\").\n\nIt is true that we build upon InstructPix2Pix [4], but InstructPix2Pix does not support visual prompting (it seems the reviewer also acknowledges this, per \"It is fascinating to see how visual cues can be translated into text-based editing directions\"?).\nWe have proposed a novel approach to enable image editing through visual prompting.\n\nRegarding your comment “except for the editing directions found in textual embedding space”: we actually see this is a major contribution, as we are the first to do so, and empirically demonstrate it can lead to clear advantages in image editing especially when the desired edit is difficult to describe with language.\n\n**Furthermore, the paper neglects to mention other related works such as SINE: SINgle Image Editing with Text-to-Image Diffusion Models (CVPR 2023) which employs style transfer for image editing through a diffusion model as seen in Figure 11 and Section 4.4 of their paper similar to image prompting. (Additionally, MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation (AAAI 2023) is also absent in the current work. It is crucial to compare and discuss these related works for a comprehensive analysis, which is currently missing in the manuscript.)**\n\nWe are happy to include these papers in the Related Work section, but note that they are only loosely related and not closely related to our work.\n\nThe setting in Section 4.4 of SINE [9] is different from our visual prompting setting. Recall that we learn an edit from a before-and-after pair, then apply it to a test image. In contrast, in Section 4.4 of SINE, there is only a test image and a reference image (style transfer based on reference image). Thus, our setting is more general than SINE. Moreover, SINE requires finetuning the diffusion model again for each edit, while ours only needs to optimize one embedding vector.\n\nLikewise, our work is also only loosely similar to the MIDMs [10] setting. MIDMs is designed for exemplar-based image translation, i.e., producing image $I_{xy}$ by combining content $I_x$ and style $I_y$. In contrast, our method aims to learn edits from before-and-after images (i.e., learning the transformation from $I_x$ to $I_y$, and then applying that learned transformation to another test image). Moreover, MIDMs framework is implemented in latent space (noise space), while ours is in text-space.\n\n**What if more examples are provided?**\n\nMore examples improve the performance, as depicted in Table 1 (Last 4 rows). Result indicates that increasing the number of example pairs (1 -> 4) increases the Directional CLIP score (0.113 -> 0.133).\n\n**What is the average time required to find a single editing direction?**\n\nIt takes roughly 7 minutes to find an editing direction. More implementation details can be found in the Supplementary material (Section C1).\n\n*Reference:*\n\n[1] Hertzmann et al., *Image analogies*, SIGGRAPH 2001.\n\n[2] Liao et al., *Visual attribute transfer through deep image analogy*, SIGGRAPH 2017.\n\n[3] Amir et al., *Visual Prompting via Image Editing*, NeurIPS 2022.\n\n[4] Tim et al., *InstructPix2Pix Learning to Follow Image Editing Instructions*, CVPR 2023.\n\n[5] Chenlin et al., *SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations*, ICLR 2022.\n\n[6] Bahjat et al., *Imagic: Text-Based Real Image Editing with Diffusion Models*, CVPR 2023.\n\n[7] Ron et al., *Null-text Inversion for Editing Real Images using Guided Diffusion Models*, CVPR 2023.\n\n[8] Rinon et al., *An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion*, ICLR 2023.\n\n[9] Zhixing et al., *SINE: SINgle Image Editing with Text-to-Image Diffusion Models*, CVPR 2023.\n\n[10] Junyong et al., *MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation*, AAAI 2023."}, {"review_id": "dVSnD0IySj", "reviewer": "Reviewer_VA7A", "summary": "- This paper proposes a novel image editing method via visual prompts\n- This paper introduces a new method to bind text-based transferring to a specific conversion between image pairs.", "questions": "- I wonder for one text prompt, how many pairs are required to train a stable directional prompt?", "limitations": "Please refer the weakness, the main limitation would be the availability of the image pairs during the real application.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow.\n- The motivation is clear and reasonable.\n- The proposed method of using image pairs to guide text directions is novel. It introduces a new method to bind text-based transferring to a specific conversion between image pairs, which can be useful.\n- The effectiveness of the method is validated.", "weaknesses": "- According to section 4.1, a key issue is that this work relies on existing pre-trained models to obtain high-quality paired images, I wonder will the quality heavily relied on the quality of these pre-trained models?\n- The method may have limited application, since the style transfer has to be built based on existing image pairs. Which means a stable style transfer model is already available, the contribution of this work is to bind the specific transfer to a text prompt. However, during the practical application, a desired style transfer may not have available paired ground-truth images.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback! We address your questions/concerns below.\n\n**According to section 4.1, a key issue is that this work relies on existing pre-trained models to obtain high-quality paired images, I wonder will the quality heavily relied on the quality of these pre-trained models?**\n\nIn practice, paired images can be provided by users. The paired examples (before-and-after) do not necessarily have to be generated by pretrained models.\n\nFor example, in Figure 1 (Row 2), it is a real roadmap and satellite image. In Figure 3 (Last row), it is a real human face and photoshopped version of it. Same with Figure 7c, before-and-after images are individually collected from the internet (e.g., [this reddit link](https://www.reddit.com/r/Frozen/comments/j4afdf/elsa_anna_kristoff_in_real_life/)). Full photo attributions can be found in the last section of the Supplementary material.\nSo the quality does not rely on pre-trained models.\n\n**The method may have limited application, since the style transfer has to be built based on existing image pairs. Which means a stable style transfer model is already available, the contribution of this work is to bind the specific transfer to a text prompt.**\n\nStyle transfer is just one application of our method. Beyond style transfer, we can also perform other image editing tasks (ie. domain translation). For example:\n- Figure 3 (Row 3): wolf <-> dog\n- Figure 6 (Row 1): dog <-> fox\n- Figure 4 (Supplementary, Row 4): add sunflowers, add roses, etc.\n- Supplementary (index.html file, Case ID: 0282472): add deer\n- Supplementary (index.html file, Case ID: 0055735): add fog\n- Supplementary (index.html file, Case ID: 0369699: replace cliff with skyscraper\n- … and more.\n\n**During the practical application, a desired style transfer may not have available paired ground-truth images.**\n\nThere are many use cases where such paired ground-truth is available.\n\nRecall our example in Introduction (Figure 1, Second row), imagine that you want to transform a roadmap image into an aerial one.  In this case, you will only have to annotate one example, and our method can help you automatically apply that transformation to new images.\n\nAnother use case is *learning your specific drawing style*. Imagine that you have spent hours, days, or months to draw your cat, in your very own style. It’s very stunning, and now you want to draw again, using the same style, but now, your dog.\nInstead of spending another hours, days, or months to start all over again, you can use our method to learn your very specific style… Then a new drawing will only be minutes away!\n\n**I wonder for one text prompt, how many pairs are required to train a stable directional prompt?**\n\nAlthough more pairs can help (as we show in Figure 2 in the Rebuttal PDF file), in general, only one before-and-after pair is sufficient to learn the edit direction (Table 1, \"Fixed noise\", Row 9-12).\n"}, {"review_id": "ShK8N6QLHP", "reviewer": "Reviewer_vo15", "summary": "This paper proposes a method for image editing via visual prompting. Given pairs of example that represent the “before”and “after” images of an edit, this framework can learn a text-based editing direction that can perform the same edit on the new images. Experimental results show the effectiveness of the proposed approach, even with one example pair.", "questions": "Please answer the questions in the weakness section.", "limitations": "The results of this paper is not sufficient, and more analysis about the parameters can further improve the contributions of this paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed framework can learn an edit direction that can be applied on new images, even with one example pair.", "weaknesses": "1.\tThe results in the first row of Fig. 3 have some artifacts in the hair and face; the results in the third row of Fig. 3 have the undesired changes in the background. Thus, the performance with only one example pair is not very good. The visual results in Fig. 4 are also not satisfactory. There should be a visual user study.\n\n\n2.\tHow to set the values of $\\lambda_{mse}$ and $\\lambda_{clip}$ for different example pairs? I wonder the robustness of the proposed methods towards different hyper-parameters. Moreover, The CLIP space cannot describe all texture changes, and we should consider the balance between the reconstruction and the CLIP guidance.\n\n\n3.\tThis framework is built based on InstructPix2Pix, which is trained with a number of “before” and “after” pairs. Thus, I wonder the performance when an example pair not in InstructPix2Pix’s training dataset is set as the condition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback! We address your questions/concerns below.\n\n**The results in the first row of Fig. 3 have some artifacts in the hair and face; the results in the third row of Fig. 3 have the undesired changes in the background.**\n\nWe admit that the background can sometimes change (as in the Fig. 3 wolf -> dog result).\nYou can notice that other baselines have background changes as well (Figure 3, Row 3, Column 5-7).\nWe find that our results can be improved by more carefully selecting the noise.\nHow to automatically select better noise is an ongoing research direction [1,2]. (Please refer to Rebuttal PDF file, Figure 3 for more qualitative result).\n\nIt is worth noting that visual prompting is demonstrated to be sensitive to example pairs too [3,4]. In Figure 3 (Row 3, Column 1-2), there is snow in the background of the before-and-after pair. Thus, the learned instruction might have learned something related to “snow” to better capture the edit. We briefly discussed it in Section 6 (Discussion, Figure 7c), as there is an open question to understand what makes a good example pair.\n\n**The performance with only one example pair is not very good. The visual results in Fig. 4 are also not satisfactory. There should be a visual user study.**\n\nWithout further clarification, we find the comments regarding Figure 4 difficult to respond to.\nFigure 4 demonstrates that by showing an example pair, our method can learn and replicate the distinctive characteristics of each specific art style (drawing, painting) for the same text instruction “Turn it into a drawing/ painting”.\nWe perceive that the learned style is visually close to the given example pair.\n\nRegarding the visual user study: we did consider conducting a user study.\nHowever, we decided against it as *we felt it is not possible to guarantee a fair comparison* between our approach and the text-conditioned baselines.\n\nFor example, in Figure 3 (Second row), the edit prompt is \"make it a watercolor painting\". Without access to an additional before-and-after image as our approach, the outputs from all baselines appear equally valid (Column 5-6), as they successfully transform the test image into a \"watercolor painting\". However, they may miss certain aspects that users want from the inference image such as red colored flowers in the example (Column 2).\n\nConversely, it is also unfair to our approach (using only the before-and-after pair) as we lack knowledge of the exact edit provided by the text prompt. Using Figure 3 (Second row) as an example once more, not knowing the specific text prompt might lead to various interpretations (e.g., \"colorize it\", \"make the coat pink\", etc.). Thus it is unclear for us how to make a fair comparison (i.e., whether we should provide a reference image or instruction during the use study).\n\n**How to set the values of $\\lambda_{clip}$ and $\\lambda_{mse}$ for different example pairs? I wonder the robustness of the proposed methods towards different hyper-parameters. Moreover, The CLIP space cannot describe all texture changes, and we should consider the balance between the reconstruction and the CLIP guidance.**\n\nWe mentioned in Section 4.1 (Line 170-173) that in all experiments, we set hyperparameters of $\\lambda_{mse}$ and $\\lambda_{clip}$ to 4 and 0.1, respectively. We find that this is generally sufficient to achieve good results, and that our method is not very sensitive to small changes in their exact values.\n\nWe do consider the image reconstruction by Image CLIP Similarity (similar to [5]), as shown in Figure 5. Image CLIP Similarity score indicates how similar the \"test\" and \"edited\" images are. In other words, it tells us how much the edited image looks different from the test image. Results show that our method performs similarly to other state-of-the-art models.\n\n**This framework is built on InstructPix2Pix, which is trained with a number of “before” and “after” pairs. Thus, I wonder about the performance when an example pair not in InstructPix2Pix’s training dataset is set as the condition.**\n\nWe indeed have many in-the-wild examples!\n\nAll images in Figure 2 and Figure 4 are in-the-wild photos (not in InstructPix2Pix dataset). Last row of Figure 3 is also in-the-wild. You can find more in-the-wild examples in the Supplementary (Figure 1, Figure 2 (Row 1, 3-7), all Figure 4).\n\n*Reference:*\n\n[1] Clinton et al., *Interpolating between Images with Diffusion Models*, arXiv 2023. (Section 4.4)\n\n[2] Bahjat et al., *Imagic: Text-Based Real Image Editing with Diffusion Models*, CVPR 2023. (Supplementary - Section B)\n\n[3] Yuanhan et al., *What Makes Good Examples for Visual In-Context Learning?*, arXiv 2023.\n\n[4] Yanpeng et al., *Exploring Effective Factors for Improving Visual In-Context Learning*, arXiv 2023.\n\n[5] Tim et al., *InstructPix2Pix Learning to Follow Image Editing Instructions*, CVPR 2023."}, {"review_id": "tZfjSeHAms", "reviewer": "Reviewer_G8X8", "summary": "The authors propose a method for finding a text-based editing direction extracted from a pair of “before” and “after” images depicting the desired edit. Using a fixed, pretrained diffusion model (in this case Stable Diffusion), the authors optimize the text-conditioning embedding to align with the CLIP-space direction between the two images. This is done will minimizing a reconstruction loss to ensure that the main details of the original image are preserved. Qualitative and quantitative results show the effectiveness of the method when compared to existing approaches (InstructPix2Pix, SDEdit, and other image editing techniques such as Imagic). The authors also provide a useful analysis of how the noise and initialization influence the generation process when aiming to edit a given image. ", "questions": "**General Comments:**   \n- The idea of the cosine direction to represent the editing direction has been used previously (e.g., StyleCLIP, StyleGAN-NADA). These works should be cited when introducing the CLIP-direction loss (Equation 3). \n\n**Questions:**   \n- Would starting from an initial prompt depicting the desired edit speed up convergence? Similarly, does the optimization process converge faster if we provide more pairs? \n- What is the significance of using InstructPix2Pix as the network? Would simply using the original Stable Diffusion model still lead to good editing results? This would be helpful to see whether the proposed scheme is robust to different diffusion models. \n- Is there any significance to the “tokens” learned in the optimized conditioning text embedding? That is, is it possible to interpret the optimized token in human-understandable tokens? E.g., does quantizing the learned embeddings to real tokens lead to a coherent text prompt? \n- How does changing the number of learnable tokens affect the result? How was the choice of using 10 tokens made? It would be very useful to further analyze this design choice with an ablation study.\n- For the quantitative evaluations (Section 4.3), the authors specify that 300 editing directions were sampled from the clean-instructpix2pix dataset. Were these prompts used in the training of InstructPix2Pix? \n", "limitations": "The authors discuss several limitations of their method including the reliance on the diffusion model and the ability to perform small edits over the image. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors present a simple, yet effective technique for extracting text-based editing directions in a pretrained diffusion model. This is particularly useful when a desired edit is difficult to describe with language. \n- Extracting this direction requires only a single exemplar pair, making it applicable and easy to use across many settings.\n- The instruction concatenation technique is very interesting and beneficial for providing users more control in cases where the extract direction is not perfect. I almost missed this, but found it to be a very interesting addition. I would consider adding this to the main paper if possible as it provides an additional advantage over alternative techniques. \n- Comparisons are performed to a variety of techniques including InstructPix2Pix and SDEdit. Further evaluations are performed on Textual Inversion, Imagic, and Null-Text Inversion. The results provided by the authors seem to out-perform existing techniques across a variety of edits and images. ", "weaknesses": "- It appears that the proposed method is more effective on style-based edits, as I could not find any examples to require a large change in the structure of the object. How does the method fair when the prompt pair depicts a change in structure (e.g., sitting/jumping)? I could not find examples or a discussion on this.\n- In Figure 7 the authors show that a single example may not generalize well to all different test sets. This is expected, but I am wondering how sensitive the method is to the choice of the exemplar pair. Assessing this could be challenging, but one possible way to assess this could be evaluating the success rate of a single input pair across a set of test images. This could be done either quantitatively (e.g., using CLIP-space similarities) or qualitatively using a user study, if possible. \n- In Table 1, the authors demonstrate that adding additional exemplars can help achieve better results. It would be great to see visual results where multiple pairs assisted in improving the results. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback! We address your questions/concerns below.\n\n**It appears that the proposed method is more effective on style-based edits, as I could not find any examples to require a large change in the structure of the object. [...]**\n\nEdits that “require a large change in the structure of the object” are typically more challenging than style-based edits. \nWe do have some examples that might answer your question:\n- Supplementary (Figure 4, Last row): Add a gun, add sunflowers, add roses.\n- Supplementary (index.html file, Case ID: 0282472): Add a herd of deer\n- Supplementary (index.html file, Case ID: 0135140): Add a thunderstorm\n- Supplementary (index.html file, Case ID: 0225198): Add a tiara\n- … and more.\n\n**[...] a single example may not generalize well to all different test sets. This is expected, but I am wondering how sensitive the method is to the choice of the exemplar pair. [...] one possible way to assess this could be evaluating the success rate of a single input pair across a set of test images. [...]**\n\nThis is a very interesting question and suggestion. We have performed an initial investigation, but do not see any clear pattern yet. We will continue to look into this and report any interesting findings in the final version.\n\n**In Table 1, the authors demonstrate that adding additional examples can help achieve better results. It would be great to see visual results [...].**\n\nThank you for your suggestion! We have created a Figure for this, please refer to Additional PDF File (Figure 2). We will also add this Figure into our Supplementary material.\n\n**Would starting from an initial prompt depicting the desired edit speed up convergence? Similarly, does the optimization process converge faster if we provide more pairs?**\n\nWe find that initialization from the ground-truth prompt hurts performance (Table 1, Row 4-8).\nWe hypothesize this may be because human-language can be mismatched with machine-preference [1].\n\nMore example pairs will help to learn the desired edits faster and more precisely. Consider the settings where we have fixed all hyper-parameters and vary only the number of examples. The results (Table 1, Rows 4-8) indicate that under the same optimization steps (N =1000 steps), having more examples (1 -> 4) boosts CLIP Directional Similarity scores (0.113 -> 0.133). These scores indicate that the learned instruction is more aligned with the desired edit.\n\n**What is the significance of using InstructPix2Pix as the network? [...]**\n\nInstructPix2Pix [4] directly builds upon a pretrained Stable Diffusion model’s vast text-to-image generation capabilities, while further finetuning it with 450,000 (text instruction, before image, after image) triplets. Thus, its learned instruction space is already rich enough to cover many image-to-image translation edits, and is therefore a good starting point for our approach.\n\nAnother reason is that the architecture of InstructPix2Pix is a natural fit for our task. Exploring how to make our approach work for general text-to-image models (e.g. Stable Diffusion) would be a good future research direction.\n\n**Is there any significance to the “tokens” learned in the optimized conditioning text embedding? That is, is it possible to interpret the optimized token in human-understandable tokens? [...]**\n\nThis is an interesting question!\nWe have tried to convert optimized tokens into natural tokens, however, we find that the natural tokens might not reflect the edit.\n\nInspired by [3], we mapped each learned tokens to their nearest tokens within the [CLIP vocabulary corpus](https://huggingface.co/timbrooks/instruct-pix2pix/raw/main/tokenizer/vocab.json).\nFor example, the human-understandable tokens in Figure 4 (Row 4, Column 2) is `ius souri anemone beans required throat alise eee`. These tokens are unrelated to the desired edit, which should be something similar to \"Turn it into a watercolor painting.\"\n\nOur findings align with previous empirical evidence [2], which suggests a disconnect between continuous prompts and their discrete interpretations. In [2], authors demonstrated that an accurate continuous prompt tailored for a specific task (e.g., describing a precise painting) could be projected into arbitrary or even irrelevant text/statements (e.g., code or a question).\n\n**How does changing the number of learnable tokens affect the result? How was the choice of using 10 tokens made? [..]**\n\nWe employ [3] to initialize tokens (Section 5 - Initialization). In [3], 8 tokens are shown to be sufficient enough to yield a stable performance. Plus 
 and 
 tokens, we have 10 tokens as initialization.\nWe find that longer prompts do not necessarily produce better results (As demonstrated in Section 4.2 of [3]).\n\n**[...] Were 300 editing directions used in the training of InstructPix2Pix?**\n\nYes, as the Clean-InstructPix2Pix dataset [4] does not come with a train/ test split.\nHowever, there are many in-the-wild examples: All images in Figure 2 and Figure 4 are in-the-wild photos (not in InstructPix2Pix dataset).\nLast row of Figure 3 is also in-the-wild. You can find further in-the-wild examples in the Supplementary (all Figure 1, Figure 2 (Row 1, 3-7), all Figure 4).\n\n**The idea of the cosine direction to represent the editing direction has been used previously (e.g., StyleCLIP, StyleGAN-NADA). These works should be cited when introducing the CLIP-direction loss (Equation 3).**\n\nThank you for pointing it out. We have updated our paper accordingly.\n\n*Reference:*\n\n[1] Yaru et al., *Optimizing Prompts for Text-to-Image Generation*, arXiv 2022.\n\n[2] Daniel et al., *Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts*, ACL 2022.\n\n[3] Yuxin et al., *Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery*, arXiv 2023.\n\n[4] Tim et al., *InstructPix2Pix Learning to Follow Image Editing Instructions*, CVPR 2023.\n"}]

Author rebuttal:
We propose a framework for *inverting visual prompts into editing instructions* for text-to-image diffusion models.
Furthermore, our method can combine instructions between learned and natural language, *yielding a hybrid editing instruction that is more precise*.


We are grateful that **all reviewers appreciate either the originality, experimentation, clarity, and/or significance** of our paper.
- **Originality**: "very interesting and important" (#uc51); "impressive" (#h5yQ), "clear and reasonable", (#VA7A), "simple, yet effective" (#G8X8), "can learn an edit ... even with one example pair." (#vo15).
- **Experimentation**: "thorough ablation study" (#uc51), "effectiveness is validated" (#VA7A).
- **Clarity**: excellent presentation (#G8X8), "well written" (#VA7A, #uc51), good (#h5yQ, #vo15).
- **Significance**: "fascinating" and "practical" (#h5yQ), "intuitive application" (#uc51), "useful" (#VA7A), "applicable and easy to use" (#G8X8).
---

We thank the reviewers’ time and effort in reviewing our paper.
We have incorporated reviewers' feedback into our revision.

We are glad that reviewer #G8X8 finds the hybrid instruction [*“very interesting and beneficial”*](https://openreview.net/forum?id=l9BsCh8ikK&noteId=tZfjSeHAms).
We will move this application into the main paper from the supplementary material as suggested.

Answers to individual reviewers are addressed below each review. Please let us know if you have any additional questions or concerns.


Meta-review: After the rebuttal, most reviewer concerns are addressed by the authors. There remains few questions:
1. If visual prompting is suitable for image editing tasks.
2. If visual prompting can be viewed as deep learning variant of image analogies
3. If it is reasonable to have a direct comparison between the proposed and the deep image analogies work.

After going over the literature and the comments made by the reviewers and the authors, the AC decide to accept this paper since
1. The debates around visual prompting vs. image analogies should not ignore the fact that this paper makes a major contribution toward image editing.
2. Practically, it is indeed hard to make a direct comparison quantitatively between the proposed and the deep image analogies work, and alternatively, the authors provide comprehensive discussion in the rebuttal.
3. The authors provide comprehensive experiments to compare with the image editing frameworks that can handle the problem setting in this paper.

Finally, the AC asks the authors to include all the discussions among the related works (in an organized way) in the revised paper.
Decision: Accept (poster)