SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thao Nguyen Yuheng Li Utkarsh Ojha Yong Jae Lee University of Wisconsin-Madison
https://thaoshibe.github.io/visii/
Figure 1: Image editing via visual prompting. Given a pair of before-and-after images of an edit, our approach (bottom) can learn and apply that edit along with the user's text prompt to enable a more accurate and intuitive image editing process compared to text-only conditioned approaches (top).
Text-conditioned image editing has emerged as a powerful tool for editing images. However, in many situations, language can be ambiguous and ineffective in describing specific image edits. When faced with such challenges, visual prompts can be a more informative and intuitive way to convey the desired edit. We present a method for image editing via visual prompting. Given example pairs that represent the "before" and "after" images of an edit, our approach learns a text-based editing direction that can be used to perform the same edit on new images. We leverage the rich, pretrained editing capabilities of text-to-image diffusion models by inverting visual prompts into editing instructions. Our results show that even with just one example pair, we can achieve competitive results compared to state-of-the-art text-conditioned image editing frameworks.
In the past few years, diffusion models [37, 38, 6, 29, 40, 8] have emerged as a powerful framework for image generation. In particular, text-to-image diffusion models can generate stunning images conditioned on a text prompt. Such models have also been developed for image editing [27, 4, 19, 30, 54, 13, 16, 39, 9, 26]; i.e., transforming an image into another based on a text specification. As these models rely on textual guidance, significant effort has been made in prompt engineering [49, 12, 48], which aims to find well-designed prompts for text-to-image generation and editing.
But what if the desired edit is difficult to describe in words? For example, describing your drawing style of your cat can be challenging to put into a sentence (Fig. 2a). Or imagine that you want to transform a roadmap image into an aerial one – it could be difficult to know what the different colored
Figure 2: Image editing with visual prompting. (a) Text-conditioned scheme (Prior work): Model takes an input image and a text prompt to perform the desired edit. (b) Visual prompting scheme (Ours): Given a pair of before-after images of an edit, we learn an implicit text-based editing instruction, and apply it to new images.
regions in the roadmap image are supposed to represent, leading to an incorrect output image. In such cases, it would be easier, and more direct, to convey the edit visually by showing a before-and-after example pair (Fig. 2b). In other words, language can be ambiguous when describing a specific image edit transformation, while visual prompts can offer a more intuitive and precise way to describe it.
Visual prompting for image editing has very recently been explored in [3, 44]. These works reformulate the problem as an image in-painting task, where an example image pair ("before" and "after") and query image are provided in a single grid-like image. The target output is inpainted (by forming the analogy, before: after = query: output). After training on a large dataset of computer vision tasks (e.g., edge detection, bounding box localization), these systems aim to perform any of those tasks during testing with in-context learning [3, 44-46] without further fine-tuning. However, while they can work reasonably well for standard computer vision tasks such as segmentation and colorization, they cannot be used for general image editing tasks since large datasets for arbitrary edits are typically unavailable.
This paper investigates image editing via visual prompting using text-to-image diffusion models. Inspired by textual inversion [10], which inverts a visual identity specified by an example image into the rich, pre-trained text embedding of a large vision-and-language model [38, 32] for text-to-image generation, we propose to invert the visual edit transformation specified by the example before-and-after image pair into a text instruction. In particular, we leverage the textual instruction space that InstructPix2Pix [4] has learned. Since InstructPix2Pix directly builds upon a pretrained Stable Diffusion model's vast text-to-image generation capabilities, while further finetuning it with 450,000 (text instruction, before image, after image) triplets, our hypothesis is that its learned instruction space is rich enough to cover many image-to-image translations (i.e., image edits), and thus can be fruitful for visual prompt based image editing. Specifically, given a pair of images representing the "before" and "after" states of an editing task, we learn the edit direction in text space by optimizing for the textual instruction that converts the "before" image into the "after" image. Once learned, this edit direction can then be applied to a new text image, together with a text prompt, facilitating precise image editing; see Figure 1.
Our contributions and main findings are: (1) We introduce a new scheme for image editing via visual prompting. (2) We propose a framework for inverting visual prompts into editing instructions for text-to-image diffusion models. (3) By conducting in-depth analyses, we share valuable insights about image editing with diffusion models; e.g., concatenating instructions between learned and natural language yields a hybrid editing instruction that is more precise; or reusing the same noise schedule in training for testing leads to a more balanced result between editing effects and faithfulness to the input image.
Text-to-image Models. Early works on text-to-image synthesis based on GANs [53, 57, 21, 50] were limited to small-scale and object-centric datasets, due to training difficulties of GANs. Auto
regressive models pioneered the use of large-scale data for text-to-image generation [33, 34, 41, 8, 52]. However, they typically suffer from high computation costs and error accumulation. An emerging trend is large-scale text-to-image diffusion models, which are the current state-of-the-art in image synthesis, offering unprecedented image fidelity and language reasoning capabilities. Research efforts have focused on improving their image quality, controllability, and expanding the type of conditional inputs [22, 9, 2, 54, 37, 38, 8, 29, 40, 51]. In the realm of image editing, diffusion models are now at the forefront, providing rich editing capabilities through text descriptions and conditional inputs. In this work, we investigate how to use visual prompts to guide image edits with diffusion models.
Image Editing. In the beginning, image editing was primarily done within the image space. Previous GAN-based approaches utilized the meaningful latent space of GANs to perform editing [18, 1, 24, 11]. The inversion technique [36, 7, 42, 35] has been used to obtain the latent features of the input image, perform editing in the latent space, and revert the output to the image space. More recently, with the support of CLIP [32], a bridge between images and texts, image editing can now be guided by text prompts [31, 11]. Recent models for text-conditioned image editing have leveraged CLIP embedding guidance and text-to-image diffusion models to achieve state-of-the-art results for a variety of edits. There are three main directions for research in this area: (1) zero-shot (exploiting the CLIP embedding directions, stochastic differential equations, or attention control) [13, 30, 26, 27, 20]; (2) optimizing text prompts and/or diffusion models [19, 39, 10, 16, 56]; and (3) fine-tuning diffusion models on a supervised dataset [4, 54]. In contrast to prior works that rely on text prompts to guide the editing, we aim to leverage visual prompts to better assist the process.
Prompt Tuning. Diffusion models have shown stirring results in text-to-image generation, but they can struggle to comprehend specific or novel concepts. Several works have focused on addressing this issue. Textual Inversion [10] learns a specialized token for new objects, which can later be plugged in with natural language to generate novel scenes. ReVersion [16] learns a specified text prompt for relation properties between two sets of images. Although a continuous prompt can be more task-specific, a discrete prompt is typically easier to manipulate by users. PEZ [48] proposes a method to discover prompts that can retrieve similar concepts of given input images. Instead of learning novel concepts for image generation, our work focuses on learning the transformation between an example pair of images that is better suited for image editing.
Visual Prompting. Since proposed in NLP [5], prompting has been adapted by computer vision researchers. Unlike traditional methods that require separate models for each downstream task, visual prompting utilizes in-context learning to solve different tasks during inference. The first application of visual prompts was proposed by [3], where an example and query image are combined to form a grid-image. The task solver fills in the missing portion, which contains the answer. They showed that the task solvers can perform effectively on several tasks with only training on a dataset of Computer Vision figures. Later, [44] and [45] expanded the framework to increase the number of tasks that can be solved. Recently, Prompt Diffusion [47] introduced a diffusion-based foundation for in-context learning. Although it shows high-quality in-context generation, a text prompt is still needed. Similar to textual prompts, not all visual prompts perform equally well. There are ongoing efforts to understand how to design a good example pair [55]. The most similar idea to visual prompting is the seminal work of Image Analogies [14] and its variants [43, 23]. However, these works rely on pixel-to-pixel correspondences between example and test images, and struggle to transfer high-level idea edits (e.g., "add a dog" or "turn it into a monkey"). Despite the success of visual prompting in solving a wide range of standard computer vision tasks, the question of whether one can use visual prompting for image editing remains unanswered.
In this section, we present our approach for enabling image editing via visual prompting. First, we provide a brief background on text-conditioned image editing diffusion models (Section 3.1). Section 3.2 and 3.3 describe how to invert visual prompts into text-based instructions. Finally, our full Visual Instruction Inversion algorithm is given in Section 3.4.
Let  $\{x,y\}$  denote the before-and-after example of an edit. Our goal is to learn a text-based edit  $c_{T}$  that captures the editing direction from  $x$  to  $y$ . Once learned,  $c_{T}$  can be applied to any new input image  $x'$ , to obtain an edited image  $y'$  that undergoes a similar transformation:  $x \to y \approx x' \to y'$ .
(a) Instruction Optimization
(b) Editing Direction
Figure 3: Our framework. (a) Given an example before-and-after image pair, we optimize the latent text instruction that converts the "before" image to the "after" image using a frozen image editing diffusion model. (b) We leverage the CLIP embedding space to help learn the editing direction. (c) Once learned, the instruction can be applied to a new image to achieve the same edit. Optionally, the user can also combine the learned instruction with a natural text prompt to create a hybrid instruction.
(c) Inference
To avoid confusion, we use only one image pair example to describe our approach. However, it is worth noting that our algorithm still holds for an arbitrary number of example pairs.
Diffusion models for image generation are trained on a sequence of gradually noisier image  $x$  over a series of timesteps  $t = 1, \dots, T$ . The goal is to learn a denoising autoencoder  $\epsilon_{\theta}$ , which predicts a denoised variant of a noisy version of  $x$  at each timestep  $t$ , commonly denoted as  $x_{t}$  [15]. Initially, this approach was implemented in pixel-space [6], but it has now been extended to the latent space for faster inference and improved quality [37]. Here, prior to the diffusion process, image  $x$  is encoded by an encoder  $\mathcal{E}$  to obtain the latent image  $z_{x}$ , which is subsequently decoded by a decoder  $\mathcal{D}$  to convert the latent image back to the image space. The objective function is defined as follows:
$$
\mathcal {L} = \mathbb {E} _ {\mathcal {E} (x), \epsilon \sim \mathcal {N} (0, 1), t} \| \epsilon - \epsilon_ {\theta} \left(z _ {x _ {t}}, t\right) \| _ {2}
$$
To enable diffusion models to take text prompts as conditional inputs, [37] introduced a domain-specific encoder  $\tau_{\theta}$  that projects text prompts to an intermediate representation  $c_{T}$ . This representation can then be inserted into the layers of the denoising network via cross-attention:
$$
\mathcal {L} = \mathbb {E} _ {\mathcal {E} (x), c _ {T}, \epsilon \sim \mathcal {N} (0, 1), t} \| \epsilon - \epsilon_ {\theta} (z _ {x _ {t}}, t, c _ {T}) \| _ {2}
$$
Conditioned on a text description  $c_{T}$ , diffusion models can synthesis stunning images. However, they are still not completely well-suited for image editing. Suppose that we want to edit image  $x$  to image  $y$ , conditioned on text prompt  $c_{T}$ . Text prompt  $c_{T}$  then needs to align with our desired edit  $x \rightarrow y$  and fully capture the visual aspects of  $x$ , which can be difficult. There are methods to help discover text prompts that can retrieve similar content [48, 28], however, they typically cannot accurately describe all aspects of the input image  $x$ . To address this challenge, the idea of adding the input image to the denoising network was proposed in [4, 40]. The input image  $x$  can then be encoded as  $c_{I} = \mathcal{E}(x)$  and concatenated to the latent image  $z_{y_t}$ , jointly guiding the editing process with text prompt  $c_{T}$ . Based on this idea, InstructPix2Pix [4] fine-tunes the text-to-image diffusion model in a supervised way to perform image editing. Its objective function is changed accordingly, as it now learns to denoise a noisy version of  $y$ , which is an edited image of  $x$  based on editing direction  $c_{T}$ :
$$
\mathcal {L} = \mathbb {E} _ {\mathcal {E} (y), c _ {T}, c _ {I}, \epsilon \sim \mathcal {N} (0, 1), t} \| \epsilon - \epsilon_ {\theta} \left(z _ {y _ {t}}, t, c _ {T}, c _ {I}\right) \| _ {2} \tag {1}
$$
3.2 Learning to Reconstruct Images
Prior textual inversion methods [10, 39, 16] all utilize an image reconstruction loss. However, their aim is to learn to capture the essence of the concept in the image so that it can be synthesized in new contexts, but not to faithfully follow pixel-level details of the input image which are required for image editing. The closest idea to ours is [19], but it needs to fine-tune the diffusion model again for each edit and input image. We instead exploit a pre-trained text-conditioned image editing model, which offers editing capabilities, while avoiding additional fine-tuning.
Given only two images  $\{x,y\}$  which represent the "before" and "after" images of an edit  $c_{T}$ , the first and foremost objective is to recover image  $y$ . We employ a pretrained text-conditioned image editing model proposed in [4], where we optimize the instruction  $c_{T}$  based on the supervised pair  $\{x,y\}$ . In
Figure 4: Instruction details. (a) Instruction Optimization: We only optimize a part of the instruction embedding  $c_{T}$ , called <ins>. (b) Hybrid instruction: During test time, we can add extra information into the learned instruction  $c_{T}$  to further guide the edit.
our case, conditional image  $c_{I}$  is the "before" image  $x$ , and target image is the "after" image  $y$ . The objective function is then adopted from Eq. 1 as:
$$
\mathcal {L} _ {m s e} = \mathbb {E} _ {\mathcal {E} (y), c _ {T}, z _ {x}, \epsilon \sim \mathcal {N} (0, 1), t} \| \epsilon - \epsilon_ {\theta} \left(z _ {y _ {t}}, t, c _ {T}, z _ {x}\right) \| _ {2} \tag {2}
$$
3.3 Learning to Perform Image Editing
If we rely only on the image reconstruction constraint (Eq. 2), we may learn a description of the edited image  $y$ , instead of the desired editing instruction. [31, 11, 30] has shown that the CLIP embedding [32] is a good indicator of the editing direction. [30] uses GPT-3 [5] to generate a set of sentences for the "before" and "after" domains of an edit; for example, cat  $\leftrightarrow$  dog. The mean difference between the CLIP embeddings of these sentences represents the text editing direction "before"  $\leftrightarrow$  "after".
In our case, we can use the difference between the CLIP embeddings of the "after" and "before" images to help learn the edit. Specifically, for an example pair  $\{x,y\}$ , we compute the image editing direction  $\Delta_{x\rightarrow y}$  as:
$$
\Delta_ {x \rightarrow y} = \mathcal {E} _ {\operatorname {c l i p}} (y) - \mathcal {E} _ {\operatorname {c l i p}} (x)
$$
We encourage the learned instruction  $c_{T}$  to be aligned with this editing direction (Figure 3b). To this end, we minimize the cosine distance between them in the CLIP embedding space:
$$
\mathcal {L} _ {\text {c l i p}} = \operatorname {c o s i n e} \left(\Delta_ {x \rightarrow y}, c _ {T}\right) \tag {3}
$$
3.4 Image Editing via Visual Prompting
Finally, given an example before-and-after image pair  $\{x,y\}$ , we formulate the visual prompting as an instruction optimization using our two constraints: Image reconstruction loss (Eq. 2) and CLIP loss (Eq. 3). We provide an illustration of our framework in training and testing in Figure 3a,c, and pseudocode in Algorithm 1. Our algorithm also holds for  $n$  example pairs  $\{(x_1,y_1),\ldots (x_n,y_n)\}$ . In this case,  $\Delta_{x\to y}$  becomes the mean difference of all examples, and at each optimization step, we randomly sample one pair  $\{x_i,y_i\}$ .
Once  $c_{T}$  is learned, we can apply it to a new image  $x_{test}$  to edit it into  $y_{test}$ . Moreover, our designed approach allows users to input extra information, enabling them to combine the learned instruction  $c_{T}$  with an additional text prompt (Figure 3c). To that end, we optimize a fixed number of tokens of  $c_{T}$  only, which provides us with the flexibility to concatenate additional information to the learned instruction during inference (Figure 4b). This allows us to achieve more fine-grained control over the resulting images, and is the final default approach.
We compare our approach against both image-editing and visual prompting frameworks, on both synthetic and real images. In Section 4.2, we present qualitative results, followed by a quantitative comparison in Section 4.3. Both quantitative and qualitative results demonstrate that our approach not only achieves competitive performance to state-of-the-art models, but also has additional merits in specific cases. Additional qualitative results can be found in the Appendix.
Algorithm 1 Visual Instruction Inversion (VISII)
1: Input: An example pair  $\{x,y\}$   
2: Pretrained denoising model  $\epsilon_{\theta}$ ; Image encoder  $\mathcal{E}$ ; CLIP encoder  $\mathcal{E}_{clip}$   
3: Number of optimization steps  $N$ ; Number of timesteps  $T$   
4: Hyperparameters  $\lambda_{clip}, \lambda_{mse}$ ; Learning rate  $\gamma$   
5: // Start optimization  
6: Initialize  $c_{T}$   
7: Encode  $z_{x} = \mathcal{E}(x)$ ;  $z_{y} = \mathcal{E}(y)$   
8: Compute  $\Delta_{x \to y} = \mathcal{E}_{\mathrm{clip}}(y) - \mathcal{E}_{\mathrm{clip}}(x)$   
9: for  $i = 1, \dots, N$  do  
10: Sample  $t \sim \mathcal{U}(0,T)$ ;  $\epsilon \sim \mathcal{N}(0,1)$   
11:  $z_{yt} \gets$  add  $\epsilon$  to  $z_{y}$  at timestep  $t$   
12:  $\hat{\epsilon} = \epsilon_{\theta}(z_{yt}, t, c_{T}, z_{x})$   
13:  $\mathcal{L} = \lambda_{mse} \| \epsilon - \hat{\epsilon} \|_2 + \lambda_{clip} (\cosine(c_{T}, \Delta_{x \to y}))$   
14: Update  $c_{T} = c_{T} - \gamma \nabla \mathcal{L}$   
15: end for  
16: Output:  $c_{T}$
Example - "as a painting by Salvador Dali" - 1234567890 Example 1234567890
Figure 5: Qualitative comparisons. Our method learns edits from example pairs and thus can produce visually closer edited images to the target example than other state-of-the-art baselines.
Training Setting. We use the frozen pretrained InstructPix2Pix [4] to optimize the instruction  $c_{T}$  for  $N = 1000$  steps,  $T = 1000$  timesteps. We use AdamW optimizer [25] with learning rate  $\gamma = 0.001$ ,  $\lambda_{mse} = 4$ , and  $\lambda_{clip} = 0.1$ . Text guidance and image guidance scores are set at their default value of 7.5 and 1.5, respectively. All experiments are conducted on a  $4 \times$  NVIDIA RTX 3090 machine.
Dataset. We randomly sampled images from the Clean-InstructPix2Pix dataset [4], which consists of synthetic paired before-after images with corresponding descriptions. In addition, we download paired photos from [17] to test the models. Since some real images do not have edited versions, we utilize [54] with manual text prompts to generate the after images with different edits.
Figure 6: A variety of edits can be performed for "Turn it into a drawing/painting" (Zoom in for details).
Evaluation Metrics. Following [4], we assess the effectiveness of our approach using the Directional CLIP similarity [11] and Image CLIP similarity metrics. However, as the CLIP directional metric does not reflect the transformation similarity between the before-after example and before-after output pair, we propose an additional metric called the Visual CLIP similarity. Specifically, we compute the cosine similarity between the before-after example pair and the before-after test pair as follows:  $s_{\text{visual}} = 1 - \text{cosine}(\Delta_{x \to y}, \Delta_{x' \to y'})$ .
Baseline Models. We compare our approach to two main categories of baselines: Image Editing and Visual Prompting. For image editing, we compare against InstructPix2Pix [4] and SDEdit [26], which are the state-of-the-art. We directly use the ground-truth editing instruction for InstructPix2Pix and after descriptions for SDEdit. For real images, we manually write instructions and descriptions for them, respectively. For visual prompting, we compare our approach against Visual Prompting [3]. The Visual Prompting code is from the author's official repository, while SDEdit and InstructPix2Pix codes are from HuggingFace.
Figure 5 presents qualitative comparisons. As can be seen, Visual Prompting [3] fails to perform the image editing task. Text-conditioned image editing frameworks, InstructPix2Pix [4] and SDEdit [26],
Figure 7: Quantitative comparison. Histogram of Image, Directional, and Visual CLIP similarity scores. Our results are comparable to state-of-the-art text-conditioned image editing frameworks.
can edit images based on the provided text prompts, but can fall short in producing edited images that are visually close to the "after" example. In contrast, our approach can learn the edit from the given example pair and apply it to new test images. For example, in wolf  $\leftrightarrow$  dog (Fig. 5, row 3), we not only achieve successful domain translation from wolf to dog, but we also preserve the color of the dog's coat. Please refer to the Appendix for more qualitative results.
Figure 6 demonstrates the advantage of visual prompting compared to text-based instruction. We can see that one text instruction can be unclear to describe specific edits. For example, the instruction "Turn it into a drawing" or "Make it a painting" can have multiple interpretations in terms of style and genres. In these cases, by showing an example pair, our method can learn and replicate the distinctive characteristics of each specific art style.
4.3 Quantitative Results
We perform quantitative evaluation of our method against two baselines: InstructPix2Pix [4] and SDEdit [26]. Since Visual Prompting was not effective in performing image editing tasks, we did not include it in our comparison. We randomly sampled 300 editing directions, resulting in a total of 1030 image pairs, from the Clean-InstructPix2Pix dataset.