SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We analyze the histograms of Image, Directional, and Visual CLIP Similarity (Figure 7). Results indicate that our method performs competitively to the baselines. In terms of Directional CLIP Similarity, InstructPix2Pix achieves the highest score, as it can make large changes the input image toward the editing instruction. Our method scores similarly to SDEdit, indicating that our approach can also perform well in learning the editing direction. Our approach is the most faithful to the input image, as reflected by the highest Image CLIP Similarity scores. Finally, for Visual CLIP Similarity, which measures the agreement between the changes in before-after example and before-after test images, our approach performs nearly identically to the two state-of-the-art models.
We next conduct an in-depth study to better understand our method. For all of the studies below, we sample 100 editing directions (resulting in 400 before-and-after pairs in total) from the Clean-InstructPix2Pix [4] dataset. We show that both the CLIP loss and instruction initialization are critical for achieving optimal performance. Additionally, we present some interesting findings regarding the effects of random noise, which can lead to variations in the output images.
Losses. We ablate the effect of each loss in Table 1. The additional CLIP loss helps improve scores in Visual and Directional CLIP Similarity [11], which reflect the editing directions. This shows that the CLIP loss encourages the learned instruction to be aligned with the target edit.
Initialization. Prior work utilizes a coarse user text prompt (e.g., "sculpture" or "a sitting dog") for textual initialization [10, 19], which can be practical, but may not always be effective. The reason is that natural text prompts can be misaligned with the model's preferred prompts [12]. We can also optimize upon a user's coarse input, however, we find that it is more effective to initialize the instruction vector  $c_{T}$  to be somewhere close to the editing target; i.e., a caption of the "after" image.
Table 1: Quantitative Analysis. We report Image, Directional, and Visual CLIP Similarity scores. Despite learning from only one example pair, our approach performs competitively to state-of-the-art image editing models. ("Direct": "Directional"; #: number of training pairs; "Init": Initialization of instruction; "GT": Ground-truth instruction; "Cap": Image captioning of "after" image.)
#	Losses	Init.	Random noise	Fixed noise
MSE	CLIP	GT	Cap.	Img ↑	Direct. ↑	Visual ↑	Img ↑	Direct. ↑	Visual ↑
ground-truth no training	0.824	0.196	0.301	-	-	-
✓	0.866	0.090	0.199	-	-	-
1	✓	✓	0.841	0.120	0.247	0.854	0.105	0.223
1	✓	✓	0.845	0.115	0.254	0.861	0.110	0.225
1	✓	✓	✓	0.838	0.131	0.231	0.852	0.102	0.236
1	✓	✓	✓	0.823	0.126	0.299	0.847	0.113	0.251
1	✓	✓	✓	0.823	0.126	0.299	0.847	0.113	0.251
2	✓	✓	✓	0.791	0.141	0.292	0.826	0.117	0.253
3	✓	✓	✓	0.780	0.148	0.283	0.805	0.132	0.256
4	✓	✓	✓	0.798	0.148	0.280	0.812	0.133	0.260
Figure 8: Fixed noise leads to more balanced results. Different noises can lead to large variations in the output. Using the same training noises yields a balanced trade-off between editing manipulation and image reconstruction.
We evaluate both initialization strategies, including user input and captioning. To mimic a user's coarse input, we directly use ground-truth editing instructions from Clean-InstructPix2Pix dataset [4]. We employ [48] to generate captions for "after" images. Results are shown in Table 1. As expected, directly using the caption as the instruction to InstructPix2Pix will not yield good results (Row 2), but initializing our model's learned instruction from the caption helps to improve the Visual and Image CLIP Similarity scores. This indicates that the learned instruction is more faithful to the input test image that we want to edit, while still retaining editing capabilities.
Noises. Text-conditioned models generate multiple variations of output images depending on the sampled noise sequence. This is true for our approach too. However, for image editing, we would prefer outputs that best reflect the edit provided in the before-and-after image pair, and preserve the test image as much as possible apart from that edit. We find that using identical noises from training in test time can help achieve this. Specifically, denote the noises sampled during the training optimization timesteps  $t = 1\ldots T$ , as  $\{\epsilon_1,\dots \epsilon_T\}$ , which are added to the latent image  $z_{y}$ . We reuse the corresponding noises in the backward process during test time to denoise the output images. This technique helps to retain the input image content, as shown in Table 1. However, there is a trade-off between aggressively moving toward the edit, and retaining the conditional input. It is worth noting that in test time, we can also use random noises for denoising if desired. We visualize this phenomenon in Figure 8.
(a) Failure Case
Before
Instruction: "add a dinosaur"
After
Learned Instruction
(c) What makes a good example?
(b) Interesting Case
Before
Test lma
Figure 9: Discussion. (a) Failure Case: Our model can fail to capture fine details. (b) Interesting case: By preparing an image and segmentation pair, we can perform image segmentation. (c) Quality of example pair: One example does not work equally well for different test images.
Hybrid instruction. Finally, our approach allows users to incorporate additional information into the learned instruction. Specifically, we create a hybrid instruction by concatenating the learned instruction with the user's text prompt. This hybrid instruction better aligns with the given example pair while still following the user's direction. In Figure 1, we transform "cat"  $\rightarrow$  "watercolor cat". We demonstrate how concatenating extra information to the learned instruction (<ins>) enables both image editing (changing to a watercolor style) and domain translation (e.g., "cat"  $\rightarrow$  "tiger"). The painting style is consistent with the before-and-after images, while the domain translation corresponds to the additional information provided by the user. Figure 12 provides more qualitative examples. Applying InstructPix2Pix [4] often does not yield satisfactory results, as the painting style differs from the reference image.
We presented a novel framework for image editing via visual prompt inversion. With just one example representing the "before" and "after" states of an image editing task, our approach achieves competitive results to state-of-the-art text-conditioned image editing models. However, there are still several limitations and open questions left for future research.
One major limitation is our reliance on a pre-trained model, InstructPix2Pix. As a result, it restricts our ability to perform editing in the full scope of diffusion models, and we might also inherit unwanted biases. Additionally, there are cases where our model fails, as shown in Figure 9a, where we fail to learn "add a dinosaur" to the input image, presumably because it is very small.
As we address the question of effectively using visual prompting with diffusion models, one might ask an interesting question in the reverse direction: Can diffusion models be used as a task solver for downstream computer vision tasks? We find out that by preparing a foreground segmentation as a green area, we can learn instructions and apply them to new images to obtain corresponding segmentations (Figure 9b). However, further research is needed to fully explore this question. We acknowledge that this question is beyond the scope of our current study, which is primarily focused on image editing. Additionally, visual in-context learning has been shown to be sensitive to prompt selection [55, 3]. Figure 9c shows cases where one example may not fit all test images. This shows that there are open questions regarding what makes a good example for image editing.
Acknowledgements. This work was supported in part by NSF CAREER IIS2150012, Adobe Data Science award, Sony Focused Research award, and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-00187238, Development of Large Korean Language Model Technology for Efficient Pretraining).
[42] Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for stylegan image manipulation. arXiv preprint arXiv:2102.02766 (2021)
[43] Šubrtová, A., Lukáč, M., Čech, J., Futschik, D., Shechtman, E., Sykora, D.: Diffusion image analogies. In: ACM SIGGRAPH 2023 Conference Proceedings. SIGGRAPH '23, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3588432.3591558, https://doi.org/10.1145/3588432.3591558
[44] Wang, X., Wang, W., Cao, Y., Shen, C., Huang, T.: Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499 (2022)
[45] Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., Huang, T.: Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 (2023)
[46] Wang, Z., Jiang, Y., Lu, Y., Shen, Y., He, P., Chen, W., Wang, Z., Zhou, M.: In-context learning unlocked for diffusion models. arXiv preprint arXiv:2305.01115 (2023), https://arxiv.org/abs/2305.01115
[47] Wang, Z., Jiang, Y., Lu, Y., Shen, Y., He, P., Chen, W., Wang, Z., Zhou, M.: In-context learning unlocked for diffusion models. arXiv preprint arXiv:2305.01115 (2023), https://arxiv.org/abs/2305.01115
[48] Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., Goldstein, T.: Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. In: arXiv (2023)
[49] Witteveen, S., Andrews, M.: Investigating prompt engineering in diffusion models. In: arXiv (2022)
[50] Xu, T., Zhang, P., Huang, Q., Han Zhang, Z.G., Huang, X., He, X.: Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In: CVPR (2018)
[51] Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., Wang, L.: Reco: Region-controlled text-to-image generation. In: arXiv (2022)
[52] Yu, J., Xu, Y., Koh, J.Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B.K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge, J., Wu, Y.: Scaling autoregressive models for content-rich text-to-image generation. In: arXiv (2022)
[53] Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.: Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: arXiv (2017)
[54] Zhang, L., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: arXiv (2023)
[55] Zhang, Y., Zhou, K., Liu, Z.: What makes good examples for visual in-context learning? In: arXiv (2023)
[56] Zhang, Z., Han, L., Ghosh, A., Metaxas, D., Ren, J.: Sine: Single image editing with text-to-image diffusion models. arXiv preprint arXiv:2212.04489 (2022)
[57] Zhu, M., Pan, P., Chen, W., Yang, Y.: Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In: arXiv (2019)
A Textual Inversion vs. Visual Instruction Inversion
Textual Inversion [10, 39] is a method to invert a visual concept into a corresponding representation in the language space. In particular, given (i) a text-to-image pre-trained model and (ii) some images describing a visual concept (e.g., a particular kind of toy; Figure 10 bottom row), Textual Inversion learns new "words" in the embedding space of the text-to-image model to represent those visual concepts. Once these "words" are learned for that concept, they can be plugged into arbitrary textual descriptions, just like other English words, which can then be used to create the target visual concept in different contexts. Instead of learning the representation for an isolated visual concept, our approach (Visual Instruction Inversion), learns the transformation from a before-and-after image pair. This learned transformation is then applied to a test image to achieve similar edit "before"  $\rightarrow$  "after".
Figure 10: Ours vs. Textual Inversion. (a) Textual Inversion inverts a visual concept or object (e.g., a particular <cat-toy>) into a word embedding. This optimized word embedding can then be combined with a textual description to generate novel scenes. (b) Our Visual Instruction Inversion learns the transformation <ins>: "before"  $\rightarrow$  "after" in a given before-and-after image pair. This learned instruction can then be applied to new test images to perform the same edit.
Applicability of Textual Inversion for image editing. Given these differences with our proposed method, we now try to see if Textual Inversion can be used for image editing. Textual Inversion can generate a "painting of a <cat-toy> in the style of Monet" by using the learned word <cat-toy> from example images (Figure 10a, Row 1). However, the synthesized images often only capture the essence of the objects, and disregard the details of the input images. As a result, textual inversion is suitable for novel scene composition, but is not effective for image editing.
On the other hand, our Visual Instruction Inversion does not learn novel token representations for objects or concepts. Instead, we learn the edit instruction from before-and-after pairs, which can be applied to any test image to obtain corresponding edits. This allows us to achieve fine-grained control over the resulting images. For example, by providing a photo of <cat-toy>, one before and one in a specific impressionist style, we learn the transformation from before to impressionist, denoted as <ins>. Once learned, this instruction can be applied to new <cat-toy> images to achieve the same impressionist painting style, without losing the fine details of the test image (Figure 10b, Row 1).
One might suggest an alternative approach to image editing using Textual Inversion, which involves learning two tokens: one for the object and another for the style (e.g., "Painting of <cat-toy> in the style of <watercolor-portraits>"). Figure 10a (Row 2) shows the results of this approach. As can be seen, Textual Inversion still often introduces significant changes that deviate from the original input image. Thus, Textual Inversion is not suitable for accurate image editing.
B Additional Qualitative Comparisons
We present qualitative comparisons with other state-of-the-art text-conditioned image editing methods, Imagic [19] and Null-text Inversion [27] (Figure 11). These methods can generate outputs based on given text prompts, such as "A watercolor painting of a cat" (Row 3). However, the outputs often do not match the given reference. The text prompts can also be ambiguous and result in unsatisfactory outputs, as illustrated by the case of "A character in a Pixar movie" (Row 1).
Another challenge is the inconsistency of text-conditioned models, where the same text prompt can produce different outputs for different test images. For example, the text prompt "A frozen waterfall" (Row 6) generates different water colors (blue vs. white) when applied to different test images (Before-and-after pair is from [27]). Our method is more consistent in this case, as the learned instruction might have learned the water color.
We use the pretrained clip-vit-large-patch14 as the CLIP Encoder in our approach. For instruction initialization [48], we set the caption length for after image to 10 tokens. However, this specific caption length does not affect the optimization algorithm. We can optimize initialization
"A woman with a marionette-like face"
"A character in a Pixar movie"
"A watercolor painting of a cat"
"A still from a western"
"A vase of tulips"
"A frozen waterfall"
"A chocolate cake"
Before
Figure 11: Additional qualitative comparisons to Magic [19] and Null-text Inversion [27]. Imagic and Null-text Inversion fail to match the reference image as they perform edits based on ambiguous text prompts (Row 1-4); or exhibit inconsistency in producing outputs for the same prompt across test images (Row 6). In contrast, our method produces visually closer edited images to the before-after pair while demonstrating improved consistency by using the learned instructions.
$\eta = 0.6$
Null-text
instructions of varying lengths (up to 77 tokens). It takes roughly 7 minutes to optimize for one edit, and 4 seconds to apply the learned instruction to new images.
Specifically, during the optimization process, we freeze the tokens representing the start of text (
), end of text (
), and all padding tokens after end of text (
). We only update the tokens inside the text prompt, called <ins> (between </startoftext> and </endoftext>) (Figure 4a).
Photo Attribution
- Elsa (Human): reddit.com/r/Frozen
- Disney characters: princess.disney.com
Toy Story characters: toystory.disney.com
Toonify faces: toonify.photos
Figure 12: Hybrid instruction. We can concatenate extra information into the learned instruction  $c_{T}$  to navigate the edit. (Zoom in for details.)
- Girl with a Pearl Earring: wikipedia/girl-with-a-pearl-earring
- Mona Lisa: wikipedia/mona-lisa
- The Princesse de Broglie: wikipedia/Princesse-de-Broglie
- Self-portrait in a Straw Hat: wikipedia/self-portrait-in-a-straw-hat
- Bo the Shiba and Mam the Cat: instagram/avoshibe
- <cat-toy> and <watercolor-portraits> concept: huggingface.co/sd-concepts-library
- Gnochi cat, waterfall, and cake images are from Imagic [19] and Null-text Inversion [27].