Title: Visual Instruction Inversion: Image Editing via Image Prompting
Venue: NeurIPS 2023
Accepted: yes
Keywords: image editing

diffusion models

visual prompting
Abstract: Text-conditioned image editing has emerged as a powerful tool for editing images.
However, in many situations, language can be ambiguous and ineffective in describing specific image edits.
When faced with such challenges, visual prompts can be a more informative and intuitive way to convey ideas.
We present a method for image editing via visual prompting.
Given pairs of example that represent the "before" and "after" images of an edit, our goal is to learn a text-based editing direction that can be used to perform the same edit on new images.
We leverage the rich, pretrained editing capabilities of text-to-image diffusion models by inverting visual prompts into editing instructions.
Our results show that with just one example pair, we can achieve competitive results compared to state-of-the-art text-conditioned image editing frameworks.

Visual Instruction Inversion: Image Editing via Visual Prompting

Abstract

Text-conditioned image editing has emerged as a powerful tool for editing images. However, in many situations, language can be ambiguous and ineffective in describing specific image edits. When faced with such challenges, visual prompts can be a more informative and intuitive way to convey the desired edit. We present a method for image editing via visual prompting. Given example pairs that represent the 'before' and 'after' images of an edit, our approach learns a text-based editing direction that can be used to perform the same edit on new images. We leverage the rich, pretrained editing capabilities of text-to-image diffusion models by inverting visual prompts into editing instructions. Our results show that even with just one example pair, we can achieve competitive results compared to state-of-the-art text-conditioned image editing frameworks.

1 Introduction

In the past few years, diffusion models [37, 38, 6, 29, 40, 8] have emerged as a powerful framework for image generation. In particular, text-to-image diffusion models can generate stunning images conditioned on a text prompt. Such models have also been developed for image editing [27, 4, 19, 30, 54, 13, 16, 39, 9, 26]; i.e., transforming an image into another based on a text specification. As these models rely on textual guidance, significant effort has been made in prompt engineering [49, 12, 48], which aims to find well-designed prompts for text-to-image generation and editing.

But what if the desired edit is difficult to describe in words? For example, describing your drawing style of your cat can be challenging to put into a sentence (Fig. 2a). Or imagine that you want to transform a roadmap image into an aerial one - it could be difficult to know what the different colored

Before

After

Query

Example

Output

Before

Output

'Make it

a drawing'

'Turn it into

an aerial photo'

(a) Prior Work: Text-conditioned image editing

(b) Our approach: 'Visual prompting' image editing

x

y

x'

y'

c

T

x'

y'

regions in the roadmap image are supposed to represent, leading to an incorrect output image. In such cases, it would be easier, and more direct, to convey the edit visually by showing a before-and-after example pair (Fig. 2b). In other words, language can be ambiguous when describing a specific image edit transformation, while visual prompts can offer a more intuitive and precise way to describe it.

Visual prompting for image editing has very recently been explored in [3, 44]. These works reformulate the problem as an image in-painting task, where an example image pair ('before' and 'after') and query image are provided in a single grid-like image. The target output is inpainted (by forming the analogy, before:after = query:output ). After training on a large dataset of computer vision tasks (e.g., edge detection, bounding box localization), these systems aim to perform any of those tasks during testing with in-context learning [3, 44-46] without further fine-tuning. However, while they can work reasonably well for standard computer vision tasks such as segmentation and colorization, they cannot be used for general image editing tasks since large datasets for arbitrary edits are typically unavailable.

This paper investigates image editing via visual prompting using text-to-image diffusion models. Inspired by textual inversion [10], which inverts a visual identity specified by an example image into the rich, pre-trained text embedding of a large vision-and-language model [38, 32] for textto-image generation, we propose to invert the visual edit transformation specified by the example before-and-after image pair into a text instruction . In particular, we leverage the textual instruction space that InstructPix2Pix [4] has learned. Since InstructPix2Pix directly builds upon a pretrained Stable Diffusion model's vast text-to-image generation capabilities, while further finetuning it with 450,000 (text instruction, before image, after image) triplets, our hypothesis is that its learned instruction space is rich enough to cover many image-to-image translations (i.e., image edits), and thus can be fruitful for visual prompt based image editing. Specifically, given a pair of images representing the 'before' and 'after' states of an editing task, we learn the edit direction in text space by optimizing for the textual instruction that converts the 'before' image into the 'after' image. Once learned, this edit direction can then be applied to a new test image, together with a text prompt, facilitating precise image editing; see Figure 1.

Our contributions and main findings are: (1) We introduce a new scheme for image editing via visual prompting. (2) We propose a framework for inverting visual prompts into editing instructions for text-to-image diffusion models. (3) By conducting in-depth analyses, we share valuable insights about image editing with diffusion models; e.g., concatenating instructions between learned and natural language yields a hybrid editing instruction that is more precise; or reusing the same noise schedule in training for testing leads to a more balanced result between editing effects and faithfulness to the input image.

2 Related Work

Text-to-image Models. Early works on text-to-image synthesis based on GANs [53, 57, 21, 50] were limited to small-scale and object-centric datasets, due to training difficulties of GANs. Auto-

regressive models pioneered the use of large-scale data for text-to-image generation [33, 34, 41, 8, 52]. However, they typically suffer from high computation costs and error accumulation. An emerging trend is large-scale text-to-image diffusion models, which are the current state-of-the-art in image synthesis, offering unprecedented image fidelity and language reasoning capabilities. Research efforts have focused on improving their image quality, controllability, and expanding the type of conditional inputs [22, 9, 2, 54, 37, 38, 8, 29, 40, 51]. In the realm of image editing, diffusion models are now at the forefront, providing rich editing capabilities through text descriptions and conditional inputs. In this work, we investigate how to use visual prompts to guide image edits with diffusion models.

Image Editing. In the beginning, image editing was primarily done within the image space. Previous GAN-based approaches utilized the meaningful latent space of GANs to perform editing [18, 1, 24, 11]. The inversion technique [36, 7, 42, 35] has been used to obtain the latent features of the input image, perform editing in the latent space, and revert the output to the image space. More recently, with the support of CLIP [32], a bridge between images and texts, image editing can now be guided by text prompts [31, 11]. Recent models for text-conditioned image editing have leveraged CLIP embedding guidance and text-to-image diffusion models to achieve state-of-the-art results for a variety of edits. There are three main directions for research in this area: (1) zero-shot (exploiting the CLIP embedding directions, stochastic differential equations, or attention control) [13, 30, 26, 27, 20]; (2) optimizing text prompts and/or diffusion models [19, 39, 10, 16, 56]; and (3) fine-tuning diffusion models on a supervised dataset [4, 54]. In contrast to prior works that rely on text prompts to guide the editing, we aim to leverage visual prompts to better assist the process.

Prompt Tuning. Diffusion models have shown stirring results in text-to-image generation, but they can struggle to comprehend specific or novel concepts. Several works have focused on addressing this issue. Textual Inversion [10] learns a specialized token for new objects, which can later be plugged in with natural language to generate novel scenes. ReVersion [16] learns a specified text prompt for relation properties between two sets of images. Although a continuous prompt can be more task-specific, a discrete prompt is typically easier to manipulate by users. PEZ [48] proposes a method to discover prompts that can retrieve similar concepts of given input images. Instead of learning novel concepts for image generation, our work focuses on learning the transformation between an example pair of images that is better suited for image editing.

Visual Prompting. Since proposed in NLP [5], prompting has been adapted by computer vision researchers. Unlike traditional methods that require separate models for each downstream task, visual prompting utilizes in-context learning to solve different tasks during inference. The first application of visual prompts was proposed by [3], where an example and query image are combined to form a grid-image. The task solver fills in the missing portion, which contains the answer. They showed that the task solvers can perform effectively on several tasks with only training on a dataset of Computer Vision figures. Later, [44] and [45] expanded the framework to increase the number of tasks that can be solved. Recently, Prompt Diffusion [47] introduced a diffusion-based foundation for in-context learning. Although it shows high-quality in-context generation, a text prompt is still needed. Similar to textual prompts, not all visual prompts perform equally well. There are ongoing efforts to understand how to design a good example pair [55]. The most similar idea to visual prompting is the seminal work of Image Analogies [14] and its variants [43, 23]. However, these works rely on pixel-to-pixel correspondences between example and test images, and struggle to transfer high-level idea edits (e.g., 'add a dog' or 'turn it into a monkey'). Despite the success of visual prompting in solving a wide range of standard computer vision tasks, the question of whether one can use visual prompting for image editing remains unanswered.

3 Framework

In this section, we present our approach for enabling image editing via visual prompting. First, we provide a brief background on text-conditioned image editing diffusion models (Section 3.1). Section 3.2 and 3.3 describe how to invert visual prompts into text-based instructions. Finally, our full Visual Instruction Inversion algorithm is given in Section 3.4.

Let { x, y } denote the before-and-after example of an edit. Our goal is to learn a text-based edit c T that captures the editing direction from x to y . Once learned, c T can be applied to any new input image x ′ , to obtain an edited image y ′ that undergoes a similar transformation: x → y ≈ x ′ → y ′ .

InstructPix2Pix

Before

🔥

ℒ

!"#

ℒ

$%&'

After

+

𝜖

InstructPix2Pix

𝜖

Test

Output

(a) Instruction Optimization

(c) Inference

c

T

θ

(b) Editing Direction

Instruction

After

Before

c

T

'As a painting'

(optional)

To avoid confusion, we use only one image pair example to describe our approach. However, it is worth noting that our algorithm still holds for an arbitrary number of example pairs.

3.1 Preliminaries

Diffusion models for image generation are trained on a sequence of gradually noisier image x over a series of timesteps t = 1 , . . . , T . The goal is to learn a denoising autoencoder ϵ θ , which predicts a denoised variant of a noisy version of x at each timestep t , commonly denoted as x t [15]. Initially, this approach was implemented in pixel-space [6], but it has now been extended to the latent space for faster inference and improved quality [37]. Here, prior to the diffusion process, image x is encoded by an encoder E to obtain the latent image z x , which is subsequently decoded by a decoder D to convert the latent image back to the image space. The objective function is defined as follows:

To enable diffusion models to take text prompts as conditional inputs, [37] introduced a domainspecific encoder τ θ that projects text prompts to an intermediate representation c T . This representation can then be inserted into the layers of the denoising network via cross-attention:

Conditioned on a text description c T , diffusion models can synthesis stunning images. However, they are still not completely well-suited for image editing. Suppose that we want to edit image x to image y , conditioned on text prompt c T . Text prompt c T then needs to align with our desired edit x → y and fully capture the visual aspects of x , which can be difficult. There are methods to help discover text prompts that can retrieve similar content [48, 28], however, they typically cannot accurately describe all aspects of the input image x . To address this challenge, the idea of adding the input image to the denoising network was proposed in [4, 40]. The input image x can then be encoded as c I = E ( x ) and concatenated to the latent image z y t , jointly guiding the editing process with text prompt c T . Based on this idea, InstructPix2Pix [4] fine-tunes the text-to-image diffusion model in a supervised way to perform image editing. Its objective function is changed accordingly, as it now learns to denoise a noisy version of y , which is an edited image of x based on editing direction c T :

3.2 Learning to Reconstruct Images

Prior textual inversion methods [10, 39, 16] all utilize an image reconstruction loss. However, their aim is to learn to capture the essence of the concept in the image so that it can be synthesized in new contexts, but not to faithfully follow pixel-level details of the input image which are required for image editing. The closest idea to ours is [19], but it needs to fine-tune the diffusion model again for each edit and input image. We instead exploit a pre-trained text-conditioned image editing model, which offers editing capabilities, while avoiding additional fine-tuning.

Given only two images { x, y } which represent the 'before' and 'after' images of an edit c T , the first and foremost objective is to recover image y . We employ a pretrained text-conditioned image editing model proposed in [4], where we optimize the instruction c T based on the supervised pair { x, y } . In

[pad]

'Turn  it  into a brown cake'







Tokenizer

[pad]




'burning candle'

<ins>

Tokenizer




Test Image

<ins>

<ins>

+

'burning candle'

<ins>

+

'with orange slices'

(b) Instruction Concatenation

(a) Instruction Optimization

After

Before

<ins>

🔥

our case, conditional image c I is the 'before' image x , and target image is the 'after' image y . The objective function is then adopted from Eq. 1 as:

3.3 Learning to Perform Image Editing

If we rely only on the image reconstruction constraint (Eq. 2), we may learn a description of the edited image y , instead of the desired editing instruction. [31, 11, 30] has shown that the CLIP embedding [32] is a good indicator of the editing direction. [30] uses GPT-3 [5] to generate a set of sentences for the 'before' and 'after' domains of an edit; for example, cat ↔ dog. The mean difference between the CLIP embeddings of these sentences represents the text editing direction 'before' ↔ 'after'.

In our case, we can use the difference between the CLIP embeddings of the 'after' and 'before' images to help learn the edit. Specifically, for an example pair { x, y } , we compute the image editing direction ∆ x → y as:

We encourage the learned instruction c T to be aligned with this editing direction (Figure 3b). To this end, we minimize the cosine distance between them in the CLIP embedding space:

3.4 Image Editing via Visual Prompting

Finally, given an example before-and-after image pair { x, y } , we formulate the visual prompting as an instruction optimization using our two constraints: Image reconstruction loss (Eq. 2) and CLIP loss (Eq. 3). We provide an illustration of our framework in training and testing in Figure 3a,c, and pseudocode in Algorithm 1. Our algorithm also holds for n example pairs { ( x 1 , y 1 ) , . . . ( x n , y n ) } . In this case, ∆ x → y becomes the mean difference of all examples, and at each optimization step, we randomly sample one pair { x i , y i } .

Once c T is learned, we can apply it to a new image x test to edit it into y test . Moreover, our designed approach allows users to input extra information, enabling them to combine the learned instruction c T with an additional text prompt (Figure 3c). To that end, we optimize a fixed number of tokens of c T only, which provides us with the flexibility to concatenate additional information to the learned instruction during inference (Figure 4b). This allows us to achieve more fine-grained control over the resulting images, and is the final default approach.

4 Evaluation

We compare our approach against both image-editing and visual prompting frameworks, on both synthetic and real images. In Section 4.2, we present qualitative results, followed by a quantitative comparison in Section 4.3. Both quantitative and qualitative results demonstrate that our approach not only achieves competitive performance to state-of-the-art models, but also has additional merits in specific cases. Additional qualitative results can be found in the Appendix.

Algorithm 1 Vis ual I nstruction I nversion ( VISII )

1: Input : An example pair { x, y }

2: Pretrained denoising model ϵ θ ; Image encoder E ; CLIP encoder E clip

3: Number of optimization steps N ; Number of timesteps T

4: Hyperparameters λ clip , λ mse ; Learning rate γ

5: // Start optimization

6: Initialize c T

7: Encode z x = E ( x ); z y = E ( y )

8: Compute ∆ x → y = E clip ( y ) -E clip ( x )

9: for i = 1 , · · · , N do

10: Sample t ∼ U (0 , T ) ; ϵ ∼ N (0 , 1)

11: z y t ← add ϵ to z y at timestep t

12: ˆ ϵ = ϵ θ ( z y t , t, c T , z x )

13: L = λ mse ∥ ϵ -ˆ ϵ ∥ 2 + λ clip ( cosine ( c T , ∆ x → y ))

14: Update c T = c T -γ ∇L

15: end for

16: Output : c T

Example

'change to a dog'

Test Image

Ours

SDEdit

InstructPix2Pix

'make it a watercolor painting'

'toonify his face'

Before

After

Vis. Prompt

▷ Initialize instruction

▷ Encode image

▷ Compute editing direction

▷ Sample timestep and noise ▷ Prepare noisy version of z y at timestep t ▷ Predict noise condition on x ▷ Compute losses

'as a painting by Salvador Dalí'

4.1 Experimental Settings

Training Setting. We use the frozen pretrained InstructPix2Pix [4] to optimize the instruction c T for N = 1000 steps, T = 1000 timesteps. We use AdamW optimizer [25] with learning rate γ = 0 . 001 , λ mse = 4 , and λ clip = 0 . 1 . Text guidance and image guidance scores are set at their default value of 7 . 5 and 1 . 5 , respectively. All experiments are conducted on a 4 × NVIDIA RTX 3090 machine.

Dataset. We randomly sampled images from the Clean-InstructPix2Pix dataset [4], which consists of synthetic paired before-after images with corresponding descriptions. In addition, we download paired photos from [17] to test the models. Since some real images do not have edited versions, we utilize [54] with manual text prompts to generate the after images with different edits.

Before

After

Before

After

Example

Example

Test Images

Test Images

Evaluation Metrics. Following [4], we assess the effectiveness of our approach using the Directional CLIP similarity [11] and Image CLIP similarity metrics. However, as the CLIP directional metric does not reflect the transformation similarity between the before-after example and before-after output pair, we propose an additional metric called the Visual CLIP similarity. Specifically, we compute the cosine similarity between the before-after example pair and the before-after test pair as follows: s visual = 1 -cosine (∆ x → y , ∆ x ′ → y ′ ) .

Baseline Models. We compare our approach to two main categories of baselines: Image Editing and Visual Prompting. For image editing, we compare against InstructPix2Pix [4] and SDEdit [26], which are the state-of-the-art. We directly use the ground-truth editing instruction for InstructPix2Pix and after descriptions for SDEdit. For real images, we manually write instructions and descriptions for them, respectively. For visual prompting, we compare our approach against Visual Prompting [3]. The Visual Prompting code is from the author's official repository, while SDEdit and InstructPix2Pix codes are from HuggingFace.

4.2 Qualitative Results

Figure 5 presents qualitative comparisons. As can be seen, Visual Prompting [3] fails to perform the image editing task. Text-conditioned image editing frameworks, InstructPix2Pix [4] and SDEdit [26],

0.4

0.6

0.8

1.0

Image CLIP Similarity

0

20

40

60

Number of images

0.1

0.0

0.1

0.2

0.3

0.4

Directional CLIP Similarity

0

10

20

30

40

50

0.0

0.2

0.4

0.6

Visual CLIP Similarity

0

10

20

30

40

50

Ours

InstructPix2Pix

SDEdit

can edit images based on the provided text prompts, but can fall short in producing edited images that are visually close to the 'after' example. In contrast, our approach can learn the edit from the given example pair and apply it to new test images. For example, in wolf ↔ dog (Fig. 5, row 3), we not only achieve successful domain translation from wolf to dog, but we also preserve the color of the dog's coat. Please refer to the Appendix for more qualitative results.

Figure 6 demonstrates the advantage of visual prompting compared to text-based instruction. We can see that one text instruction can be unclear to describe specific edits. For example, the instruction 'Turn it into a drawing' or 'Make it a painting' can have multiple interpretations in terms of style and genres. In these cases, by showing an example pair, our method can learn and replicate the distinctive characteristics of each specific art style.

4.3 Quantitative Results

We perform quantitative evaluation of our method against two baselines: InstructPix2Pix [4] and SDEdit [26]. Since Visual Prompting was not effective in performing image editing tasks, we did not include it in our comparison. We randomly sampled 300 editing directions, resulting in a total of 1030 image pairs, from the Clean-InstructPix2Pix dataset.

We analyze the histograms of Image, Directional, and Visual CLIP Similarity (Figure 7). Results indicate that our method performs competitively to the baselines. In terms of Directional CLIP Similarity, InstructPix2Pix achieves the highest score, as it can make large changes the input image toward the editing instruction. Our method scores similarly to SDEdit, indicating that our approach can also perform well in learning the editing direction. Our approach is the most faithful to the input image, as reflected by the highest Image CLIP Similarity scores. Finally, for Visual CLIP Similarity, which measures the agreement between the changes in before-after example and before-after test images, our approach performs nearly identically to the two state-of-the-art models.

5 Analysis

We next conduct an in-depth study to better understand our method. For all of the studies below, we sample 100 editing directions (resulting in 400 before-and-after pairs in total) from the CleanInstructPix2Pix [4] dataset. We show that both the CLIP loss and instruction initialization are critical for achieving optimal performance. Additionally, we present some interesting findings regarding the effects of random noise, which can lead to variations in the output images.

Losses. We ablate the effect of each loss in Table 1. The additional CLIP loss helps improve scores in Visual and Directional CLIP Similarity [11], which reflect the editing directions. This shows that the CLIP loss encourages the learned instruction to be aligned with the target edit.

Initialization. Prior work utilizes a coarse user text prompt (e.g., 'sculpture' or 'a sitting dog') for textual initialization [10, 19], which can be practical, but may not always be effective. The reason is that natural text prompts can be misaligned with the model's preferred prompts [12]. We can also optimize upon a user's coarse input, however, we find that it is more effective to initialize the instruction vector c T to be somewhere close to the editing target; i.e., a caption of the 'after' image.

Example

Test

Fixed

Random

We evaluate both initialization strategies, including user input and captioning. To mimic a user's coarse input, we directly use ground-truth editing instructions from Clean-InstructPix2Pix dataset [4]. We employ [48] to generate captions for 'after' images. Results are shown in Table 1. As expected, directly using the caption as the instruction to InstructPix2Pix will not yield good results (Row 2), but initializing our model's learned instruction from the caption helps to improve the Visual and Image CLIP Similarity scores. This indicates that the learned instruction is more faithful to the input test image that we want to edit, while still retaining editing capabilities.

Noises. Text-conditioned models generate multiple variations of output images depending on the sampled noise sequence. This is true for our approach too. However, for image editing, we would prefer outputs that best reflect the edit provided in the before-and-after image pair, and preserve the test image as much as possible apart from that edit. We find that using identical noises from training in test time can help achieve this. Specifically, denote the noises sampled during the training optimization timesteps t = 1 . . . T , as { ϵ 1 , . . . ϵ T } , which are added to the latent image z y . We reuse the corresponding noises in the backward process during test time to denoise the output images. This technique helps to retain the input image content, as shown in Table 1. However, there is a trade-off between aggressively moving toward the edit, and retaining the conditional input. It is worth noting that in test time, we can also use random noises for denoising if desired. We visualize this phenomenon in Figure 8.

Before

After

Learned Instruction

Instruction: 'add a dinosaur'

Before

After

Test Image

Output

(a) Failure Case

(b) Interesting Case

Before

After

Before

After

Example

✓

Test Images

✓

✗

✗

(c) What makes a good example?

Hybrid instruction. Finally, our approach allows users to incorporate additional information into the learned instruction. Specifically, we create a hybrid instruction by concatenating the learned instruction with the user's text prompt. This hybrid instruction better aligns with the given example pair while still following the user's direction. In Figure 1, we transform 'cat' → 'watercolor cat'. We demonstrate how concatenating extra information to the learned instruction ( <ins> ) enables both image editing (changing to a watercolor style) and domain translation (e.g., 'cat' → 'tiger'). The painting style is consistent with the before-and-after images, while the domain translation corresponds to the additional information provided by the user. Figure 12 provides more qualitative examples. Applying InstructPix2Pix [4] often does not yield satisfactory results, as the painting style differs from the reference image.

6 Discussion and Conclusion

We presented a novel framework for image editing via visual prompt inversion. With just one example representing the 'before' and 'after' states of an image editing task, our approach achieves competitive results to state-of-the-art text-conditioned image editing models. However, there are still several limitations and open questions left for future research.

One major limitation is our reliance on a pre-trained model, InstructPix2Pix. As a result, it restricts our ability to perform editing in the full scope of diffusion models, and we might also inherit unwanted biases. Additionally, there are cases where our model fails, as shown in Figure 9a, where we fail to learn 'add a dinosaur' to the input image, presumably because it is very small.

As we address the question of effectively using visual prompting with diffusion models, one might ask an interesting question in the reverse direction: Can diffusion models be used as a task solver for downstream computer vision tasks? We find out that by preparing a foreground segmentation as a green area, we can learn instructions and apply them to new images to obtain corresponding segmentations (Figure 9b). However, further research is needed to fully explore this question. We acknowledge that this question is beyond the scope of our current study, which is primarily focused on image editing. Additionally, visual in-context learning has been shown to be sensitive to prompt selection [55, 3]. Figure 9c shows cases where one example may not fit all test images. This shows that there are open questions regarding what makes a good example for image editing.

Acknowledgements. This work was supported in part by NSF CAREER IIS2150012, Adobe Data Science award, Sony Focused Research award, and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-00187238, Development of Large Korean Language Model Technology for Efficient Pretraining).

References

Alaluf, Y., Patashnik, O., Wu, Z., Zamir, A., Shechtman, E., Lischinski, D., Cohen-Or, D.: Third time's the charm? image and video editing with stylegan3. In: arXiv (2022)

Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., Karras, T., Liu, M.Y.: ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022)

Bar, A., Gandelsman, Y., Darrell, T., Globerson, A., Efros, A.A.: Visual prompting via image inpainting. arXiv preprint arXiv:2209.00647 (2022)

Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: arXiv (2023)

Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: arXiv (2020)

Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. In: Advances in Neural Information Processing Systems. vol. 34, pp. 8780-8794. Curran Associates, Inc. (2021), https://proceedings. neurips.cc/paper/2021/file/49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf

Dinh, T.M., Tran, A.T., Nguyen, R., Hua, B.S.: Hyperinverter: Improving stylegan inversion via hypernetwork. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: arXiv (2021)

Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., Taigman, Y.: Make-a-scene: Scene-based text-to-image generation with human priors. In: arXiv (2022)

Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. In: arXiv (2022). https://doi.org/10.48550/ARXIV.2208.01618, https://arxiv.org/abs/2208.01618

Gal, R., Patashnik, O., Maron, H., Chechik, G., Cohen-Or, D.: Stylegan-nada: Clip-guided domain adaptation of image generators. In: arXiv (2021)

Hao, Y., Chi, Z., Dong, L., Wei, F.: Optimizing prompts for text-to-image generation. In: arXiv (2022)

Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. In: arXiv (2022)

Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. p. 327-340. SIGGRAPH '01, Association for Computing Machinery, New York, NY, USA (2001). https://doi.org/10.1145/383259.383295, https://doi.org/10.1145/383259.383295

Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: arXiv (2020)

Huang, Z., Wu, T., Jiang, Y., Chan, K.C., Liu, Z.: ReVersion: Diffusion-based relation inversion from images. arXiv preprint arXiv:2303.13495 (2023)

Isola, P., Zhu, J.Y ., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. CVPR (2017)

Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)

Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-based real image editing with diffusion models. In: Conference on Computer Vision and Pattern Recognition 2023 (2023)

Kim, G., Kwon, T., Ye, J.C.: Diffusionclip: Text-guided diffusion models for robust image manipulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2426-2435 (June 2022)

Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R.S., Torralba, A., Urtasun, R., Fidler, S.: Skip-thought vectors. arXiv preprint arXiv:1506.06726 (2015)

Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y .J.: Gligen: Open-set grounded text-to-image generation. In: arXiv:2301.07093 (2023)

Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual attribute transfer through deep image analogy (2017)

Liu, Y., Gal, R., Bermano, A.H., Chen, B., Cohen-Or, D.: Self-conditioned generative adversarial networks for image editing. In: arXiv (2022)

Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: arXiv (2019)

Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: Sdedit: Guided image synthesis and editing with stochastic differential equations. In: arXiv (2022)

Mokady, R., Hertz, A., Aberman, K., Pritch, Y., Cohen-Or, D.: Null-text inversion for editing real images using guided diffusion models. In: arXiv (2022)

Mokady, R., Hertz, A., Bermano, A.H.: Clipcap: Clip prefix for image captioning. In: arXiv (2021)

Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In: arXiv (2022)

Parmar, G., Singh, K.K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-shot image-to-image translation. In: arXiv (2023)

Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., Lischinski, D.: Styleclip: Text-driven manipulation of stylegan imagery. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 2085-2094 (October 2021)

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: arXiv (2021)

Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. In: arXiv (2022). https://doi.org/10.48550/ARXIV.2204.06125, https://arxiv.org/ abs/2204.06125

Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. In: arXiv (2022)

Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., Cohen-Or, D.: Encoding in style: a stylegan encoder for image-to-image translation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2021)

Roich, D., Mokady, R., Bermano, A.H., Cohen-Or, D.: Pivotal tuning for latent-based editing of real images. In: arXiv (2021)

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10684-10695 (June 2022)

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: arXiv (2021)

Ruiz, N., Li, Y., Jampani, V., Pritch, Y ., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-toimage diffusion models for subject-driven generation. In: arXiv (2022)

Saharia, C., Chan, W., Chang, H., Lee, C.A., Ho, J., Salimans, T., Fleet, D.J., Norouzi, M.: Palette: Image-to-image diffusion models. In: arXiv (2022)

Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic text-to-image diffusion models with deep language understanding. In: arXiv (2022)

Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for stylegan image manipulation. arXiv preprint arXiv:2102.02766 (2021)

Šubrtová, A., Lukáˇ c, M., ˇ Cech, J., Futschik, D., Shechtman, E., Sýkora, D.: Diffusion image analogies. In: ACM SIGGRAPH 2023 Conference Proceedings. SIGGRAPH '23, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3588432.3591558, https://doi.org/10.1145/ 3588432.3591558

Wang, X., Wang, W., Cao, Y., Shen, C., Huang, T.: Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499 (2022)

Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., Huang, T.: Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 (2023)

Wang, Z., Jiang, Y., Lu, Y., Shen, Y., He, P., Chen, W., Wang, Z., Zhou, M.: In-context learning unlocked for diffusion models. arXiv preprint arXiv:2305.01115 (2023), https://arxiv.org/abs/2305.01115

Wang, Z., Jiang, Y., Lu, Y., Shen, Y., He, P., Chen, W., Wang, Z., Zhou, M.: In-context learning unlocked for diffusion models. arXiv preprint arXiv:2305.01115 (2023), https://arxiv.org/abs/2305.01115

Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., Goldstein, T.: Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. In: arXiv (2023)

Witteveen, S., Andrews, M.: Investigating prompt engineering in diffusion models. In: arXiv (2022)

Xu, T., Zhang, P., Huang, Q., Han Zhang, Z.G., Huang, X., He, X.: Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In: CVPR (2018)

Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., Wang, L.: Reco: Region-controlled text-to-image generation. In: arXiv (2022)

Yu, J., Xu, Y., Koh, J.Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B.K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge, J., Wu, Y.: Scaling autoregressive models for content-rich text-to-image generation. In: arXiv (2022)

Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.: Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: arXiv (2017)

Zhang, L., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: arXiv (2023)

Zhang, Y., Zhou, K., Liu, Z.: What makes good examples for visual in-context learning? In: arXiv (2023)

Zhang, Z., Han, L., Ghosh, A., Metaxas, D., Ren, J.: Sine: Single image editing with text-to-image diffusion models. arXiv preprint arXiv:2212.04489 (2022)

Zhu, M., Pan, P., Chen, W., Yang, Y.: Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In: arXiv (2019)

Appendix

A Textual Inversion vs. Visual Instruction Inversion

Textual Inversion [10, 39] is a method to invert a visual concept into a corresponding representation in the language space. In particular, given (i) a text-to-image pre-trained model and (ii) some images describing a visual concept (e.g., a particular kind of toy; Figure 10 bottom row), Textual Inversion learns new 'words' in the embedding space of the text-to-image model to represent those visual concepts. Once these 'words' are learned for that concept, they can be plugged into arbitrary textual descriptions, just like other English words, which can then be used to create the target visual concept in different contexts. Instead of learning the representation for an isolated visual concept, our approach (Visual Instruction Inversion), learns the transformation from a before-and-after image pair. This learned transformation is then applied to a test image to achieve similar edit 'before' → 'after'.

<

cat-toy

>

<

watercolor-portraits

>

…

'Painting of

<cat-toy>

in the style of

<watercolor-portraits>

'

Test Image

Ours

Example

Before

After

<ins>

<ins>

<ins>

<ins>

'Painting of

<cat-toy>

in the style of Monet'

🔥

🔥

🔥

🔥

(b) Visual Instruction Inversion

(a) Textual Inversion

Applicability of Textual Inversion for image editing. Given these differences with our proposed method, we now try to see if Textual Inversion can be used for image editing. Textual Inversion can generate a 'painting of a <cat-toy> in the style of Monet' by using the learned word <cat-toy> from example images (Figure 10a, Row 1). However, the synthesized images often only capture the essence of the objects, and disregard the details of the input images. As a result, textual inversion is suitable for novel scene composition, but is not effective for image editing.

On the other hand, our Visual Instruction Inversion does not learn novel token representations for objects or concepts. Instead, we learn the edit instruction from before-and-after pairs, which can be applied to any test image to obtain corresponding edits. This allows us to achieve fine-grained control over the resulting images. For example, by providing a photo of <cat-toy> , one before and one in a specific impressionist style, we learn the transformation from before to impressionist, denoted as <ins> . Once learned, this instruction can be applied to new <cat-toy> images to achieve the same impressionist painting style, without losing the fine details of the test image (Figure 10b, Row 1).

One might suggest an alternative approach to image editing using Textual Inversion, which involves learning two tokens: one for the object and another for the style (e.g., 'Painting of <cat-toy> in the style of <watercolor-portraits>' ). Figure 10a (Row 2) shows the results of this approach. As can be seen, Textual Inversion still often introduces significant changes that deviate from the original input image. Thus, Textual Inversion is not suitable for accurate image editing.

B Additional Qualitative Comparisons

Wepresent qualitative comparisons with other state-of-the-art text-conditioned image editing methods, Imagic [19] and Null-text Inversion [27] (Figure 11). These methods can generate outputs based on given text prompts, such as 'A watercolor painting of a cat' (Row 3). However, the outputs often do not match the given reference. The text prompts can also be ambiguous and result in unsatisfactory outputs, as illustrated by the case of 'A character in a Pixar movie' (Row 1).

Another challenge is the inconsistency of text-conditioned models, where the same text prompt can produce different outputs for different test images. For example, the text prompt 'A frozen waterfall' (Row 6) generates different water colors (blue vs. white) when applied to different test images (Before-and-after pair is from [27]). Our method is more consistent in this case, as the learned instruction might have learned the water color.

C Implementation Details

We use the pretrained clip-vit-large-patch14 as the CLIP Encoder in our approach. For instruction initialization [48], we set the caption length for after image to 10 tokens. However, this specific caption length does not affect the optimization algorithm. We can optimize initialization

'A character in a Pixar movie'

'A woman with a marionette-like face'

'A watercolor painting of a cat'

'A still from a western'

'A vase of tulips'

'A frozen waterfall'

'A chocolate cake'

Before

After

Test

Ours

Null-text

𝜂 = 0.6

0.8

Example

（

E

C

instructions of varying lengths (up to 77 tokens). It takes roughly 7 minutes to optimize for one edit, and 4 seconds to apply the learned instruction to new images.

Specifically, during the optimization process, we freeze the tokens representing the start of text ( 
 ), end of text ( 
 ), and all padding tokens after end of text ( 
 ). We only update the tokens inside the text prompt, called <ins> (between 
 and 
 ) (Figure 4a).

Photo Attribution

Elsa (Human): reddit.com/r/Frozen

Disney characters: princess.disney.com

Toy Story characters: toystory.disney.com

Toonify faces: toonify.photos

Test Image

Before

After

<ins>

+

'…, origami'

+ '

…, with

butterflies'

+

'…, tulip'

+

'…, marigold'

+

'Turn it into an

impressionist painting'

Instruction

Instruction

InstructPix2Pix

Test Image

<ins>

+

'…, female'

+

'..., female

with sunflowers'

+

'…, female

with roses'

+

'…, Asian'

+

'…, female

with a gun'

Test Image

Before

After

'Turn it into

a still from a

western

'

Instruction

Test Image

Ours

InstructPix2Pix

+

'…, female'

+

'..., female

with sunflowers'

+

'…, female

with roses'

+

'…, Asian'

+

'…, female

with a gun'

Instruction

Visual Prompting

Text-conditioned

Visual Prompting

Text-conditioned

Ours

+

'…, origami'

+ '

…, with

butterflies'

+

'…, tulip'

+

'…, marigold'

+

Girl with a Pearl Earring: wikipedia/girl-with-a-pearl-earring

Mona Lisa: wikipedia/mona-lisa

The Princesse de Broglie: wikipedia/Princesse-de-Broglie

Self-portrait in a Straw Hat: wikipedia/self-portrait-in-a-straw-hat

Bo the Shiba and Mam the Cat: instagram/avoshibe

<cat-toy> and <watercolor-portraits> concept: huggingface.co/sd-concepts-library

Gnochi cat, waterfall, and cake images are from Imagic [19] and Null-text Inversion [27].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for finding a text-based editing direction extracted from a pair of “before” and “after” images depicting the desired edit. Using a fixed, pretrained diffusion model (in this case Stable Diffusion), the authors optimize the text-conditioning embedding to align with the CLIP-space direction between the two images. This is done will minimizing a reconstruction loss to ensure that the main details of the original image are preserved. Qualitative and quantitative results show the effectiveness of the method when compared to existing approaches (InstructPix2Pix, SDEdit, and other image editing techniques such as Imagic). The authors also provide a useful analysis of how the noise and initialization influence the generation process when aiming to edit a given image.\n\nSTRENGTHS:\n- The authors present a simple, yet effective technique for extracting text-based editing directions in a pretrained diffusion model. This is particularly useful when a desired edit is difficult to describe with language. \n- Extracting this direction requires only a single exemplar pair, making it applicable and easy to use across many settings.\n- The instruction concatenation technique is very interesting and beneficial for providing users more control in cases where the extract direction is not perfect. I almost missed this, but found it to be a very interesting addition. I would consider adding this to the main paper if possible as it provides an additional advantage over alternative techniques. \n- Comparisons are performed to a variety of techniques including InstructPix2Pix and SDEdit. Further evaluations are performed on Textual Inversion, Imagic, and Null-Text Inversion. The results provided by the authors seem to out-perform existing techniques across a variety of edits and images.\n\nWEAKNESSES:\n- It appears that the proposed method is more effective on style-based edits, as I could not find any examples to require a large change in the structure of the object. How does the method fair when the prompt pair depicts a change in structure (e.g., sitting/jumping)? I could not find examples or a discussion on this.\n- In Figure 7 the authors show that a single example may not generalize well to all different test sets. This is expected, but I am wondering how sensitive the method is to the choice of the exemplar pair. Assessing this could be challenging, but one possible way to assess this could be evaluating the success rate of a single input pair across a set of test images. This could be done either quantitatively (e.g., using CLIP-space similarities) or qualitatively using a user study, if possible. \n- In Table 1, the authors demonstrate that adding additional exemplars can help achieve better results. It would be great to see visual results where multiple pairs assisted in improving the results.\n\nQUESTIONS:\n**General Comments:**   \n- The idea of the cosine direction to represent the editing direction has been used previously (e.g., StyleCLIP, StyleGAN-NADA). These works should be cited when introducing the CLIP-direction loss (Equation 3). \n\n**Questions:**   \n- Would starting from an initial prompt depicting the desired edit speed up convergence? Similarly, does the optimization process converge faster if we provide more pairs? \n- What is the significance of using InstructPix2Pix as the network? Would simply using the original Stable Diffusion model still lead to good editing results? This would be helpful to see whether the proposed scheme is robust to different diffusion models. \n- Is there any significance to the “tokens” learned in the optimized conditioning text embedding? That is, is it possible to interpret the optimized token in human-understandable tokens? E.g., does quantizing the learned embeddings to real tokens lead to a coherent text prompt? \n- How does changing the number of learnable tokens affect the result? How was the choice of using 10 tokens made? It would be very useful to further analyze this design choice with an ablation study.\n- For the quantitative evaluations (Section 4.3), the authors specify that 300 editing directions were sampled from the clean-instructpix2pix dataset. Were these prompts used in the training of InstructPix2Pix?\n\nLIMITATIONS:\nThe authors discuss several limitations of their method including the reliance on the diffusion model and the ability to perform small edits over the image.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

comments: SUMMARY:
This paper proposes a method for image editing via visual prompting. Given pairs of example that represent the “before”and “after” images of an edit, this framework can learn a text-based editing direction that can perform the same edit on the new images. Experimental results show the effectiveness of the proposed approach, even with one example pair.

STRENGTHS:
The proposed framework can learn an edit direction that can be applied on new images, even with one example pair.

WEAKNESSES:
1.	The results in the first row of Fig. 3 have some artifacts in the hair and face; the results in the third row of Fig. 3 have the undesired changes in the background. Thus, the performance with only one example pair is not very good. The visual results in Fig. 4 are also not satisfactory. There should be a visual user study.


2.	How to set the values of $\lambda_{mse}$ and $\lambda_{clip}$ for different example pairs? I wonder the robustness of the proposed methods towards different hyper-parameters. Moreover, The CLIP space cannot describe all texture changes, and we should consider the balance between the reconstruction and the CLIP guidance.


3.	This framework is built based on InstructPix2Pix, which is trained with a number of “before” and “after” pairs. Thus, I wonder the performance when an example pair not in InstructPix2Pix’s training dataset is set as the condition.

QUESTIONS:
Please answer the questions in the weakness section.

LIMITATIONS:
The results of this paper is not sufficient, and more analysis about the parameters can further improve the contributions of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-28

comments: SUMMARY:
- This paper proposes a novel image editing method via visual prompts
- This paper introduces a new method to bind text-based transferring to a specific conversion between image pairs.

STRENGTHS:
- The paper is well written and easy to follow.
- The motivation is clear and reasonable.
- The proposed method of using image pairs to guide text directions is novel. It introduces a new method to bind text-based transferring to a specific conversion between image pairs, which can be useful.
- The effectiveness of the method is validated.

WEAKNESSES:
- According to section 4.1, a key issue is that this work relies on existing pre-trained models to obtain high-quality paired images, I wonder will the quality heavily relied on the quality of these pre-trained models?
- The method may have limited application, since the style transfer has to be built based on existing image pairs. Which means a stable style transfer model is already available, the contribution of this work is to bind the specific transfer to a text prompt. However, during the practical application, a desired style transfer may not have available paired ground-truth images.

QUESTIONS:
- I wonder for one text prompt, how many pairs are required to train a stable directional prompt?

LIMITATIONS:
Please refer the weakness, the main limitation would be the availability of the image pairs during the real application.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates image editing via visual prompting useful when textual descriptions cannot describe desired edits. The proposed framework inverts visual prompts into editing instructions and learns directions in the text space of the pretrained instruct pix-to-pix model. This edit direction is learned from a pair of query and target images to generate desired image editing. Results suggest, one example is sufficient to learn such directions.\n\nSTRENGTHS:\n- It is fascinating to see how visual cues can be translated into text-based editing directions, especially when the edits are difficult to convey through written instructions alone. \n- The editing approach that is learned can be used to modify new test images with impressive accuracy. It is impressive that comparable results can be achieved even with just one example pair. \n- Practical insights into image editing with diffusion models, including the potential to apply the same noise schedule for both training and testing is also interesting\n\nWEAKNESSES:\nThe paper has a significant weakness in that it fails to acknowledge seminal work on image analogies, specifically, the research conducted by Hertzmann et al. (2001) and their deep learning adaptation, deep image analogies (Liao et al., 2017). While the proposed approach shares similarities with the concept of image analogies, the use of instruct pix to pix with a similar analogy lacks novelty, except for the editing directions found in textual embedding space. \n\nFurthermore, the paper neglects to mention other related works such as SINE: SINgle Image Editing with Text-to-Image Diffusion Models (CVPR 2023) which employs style transfer for image editing through a diffusion model as seen in Figure 11 and Section 4.4 of their paper similar to image prompting. Additionally, MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation (AAAI 2023) is also absent in the current work. It is crucial to compare and discuss these related works for a comprehensive analysis, which is currently missing in the manuscript.\n\nQUESTIONS:\nIs it possible to enhance the quality of editing directions by providing more examples (before and after images)? Also, what is the average time required to find a single editing direction?\n\nLIMITATIONS:\nThe paper needs to properly acknowledge previous work on image analogies and lacks originality in its approach. Other related works should be mentioned and compared for a comprehensive analysis, which is currently lacking.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new method that can perform visual prompting via a pair of exemplar images through a pretrained text-based instruction image editing model. The method introduced in this paper only requires optimizing over the text conditioning vector in order to perform visual instruction editing. The authors have shown strong qualitative results demonstrating various application scenarios.\n\nSTRENGTHS:\nThis model introduced in this paper solves a very interesting and important application problem that is exemplar-based visual prompting of image generative models. The paper is very well written and the results look very promising and this model can be easily directly applied in a large range of intuitive applications by daily users. It also leverages a pretrained model and only requires optimizing a vector, which makes it more appealing in applications. The authors have also performed thorough ablation study.\n\nWEAKNESSES:\n1. The quantitative evaluations are not very comprehensive. The authors only sampled 1k images to perform quantitative analysis, and did not report any fidelity scores. For editing controllability, they only report CLIP related scores, which is not very representative in many circumstances [1].\n2. When explaining the method, the authors sometimes mix the prior work with their contributions. A prime example would be Section 3.2, which is redundant because it is the same as InstructPix2Pix, but sectioning in this way makes it look like a new contribution at the first glance. I would recommend the authors to make clear distinctions between their contributions and the prior literature.\n\n[1] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, Candace Ross. “Winoground: Probing vision and language models for visio-linguistic compositionality”. CVPR 2022. https://arxiv.org/pdf/2204.03162.pdf\n\nQUESTIONS:\n1. How similar does the exemplar visual prompt and query image should look? Can the authors give some additional ablation study on the similarity between the visual prompt and the query images?\n2. How long does it take to sample one image?\n3. Since CLIP is not a perfect encoder and has various known limitations, I am wondering how well this method handles tasks such as color changes or duplicating objects.\n4. Related to Weakness (1), I think it is also possible to report more standardized metrics such as KID (fidelity score for small number of samples), IoU and (masked) LPIPS with tasks like semantic segmentation map to image and image compositing. Can the authors report some standardized metrics for some of these tasks?\n5. Why do more examples hurt the performance (according to Table 1)?\n\n~~I am happy to raise my score if the authors can add experiments to address my concerns.~~\n\n***After the rebuttal discussion, I would like to raise my score from 5 to 6.***\n\nLIMITATIONS:\nAlthough the imitations are extensively discussed in the paper, the discussion of potential ethical issues is missing. Since this paper uses a pretrained large image generative model, which is known to have various societal issues, I would highly recommend the authors to include a broader impact statement in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}