ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation
Authors: Yasheng SUN
Yifan Yang
Houwen Peng
Yifei Shen
Yuqing Yang
Han Hu
Lili Qiu
Hideki Koike
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Manipulation
Visual Instruction

Abstract: While language-guided image manipulation has made remarkable progress, the challenge of how to instruct the manipulation process faithfully reflecting human intentions persists. An accurate and comprehensive description of a manipulation task using natural language is laborious and sometimes even impossible, primarily due to the inherent uncertainty and ambiguity present in linguistic expressions. 
Is it feasible to accomplish image manipulation without resorting to external cross-modal language information? If this possibility exists, the inherent modality gap would be effortlessly eliminated. In this paper, we propose a novel  manipulation methodology, dubbed ImageBrush, that learns visual instructions for more accurate image editing.
Our key idea is to employ a pair of transformation images as visual instructions, which not only precisely captures human intention but also facilitates accessibility in real-world scenarios. Capturing visual instructions is particularly challenging because it involves extracting the underlying intentions solely from visual demonstrations and then applying this operation to a new image. To address this challenge, we formulate visual instruction learning as a diffusion-based inpainting problem, where the contextual information is fully exploited through an iterative process of generation. A visual prompting encoder is carefully devised to enhance the model's capacity in uncovering human intent behind the visual instructions. Extensive experiments show that our method generates engaging manipulation results conforming to the transformations entailed in demonstrations. Moreover, our model exhibits robust generalization capabilities on various downstream tasks such as pose transfer, image translation and video inpainting.

ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation
Yasheng Sun"
Yifan Yang'
Houwen Peng
Tokyo Institute of Technology
Microsoft
Microsoft
sun y aj @m. titech. ac.
yifanyangmmicrosoft..oo com
houwen. wwwpppennnnnooorr.oooooppooooeoe com
Yifei Shen
Yuqing Yang
Han Hu Microsoft
Microsoft yifeeiisennmiccosotti com yuqing www.gaannggnisseoo..o com
Microsoft
hanhu@microsoft.ccm
Lili Qiu
Hideki Koike
Microsoft
Tokyo Institute of Technology koikeêc. titech, ac jp
1il liliqiuemi crosoft...... com
Iustation
Que hzas
Duppi:
Duntuctio
Query Tragg
OuEzii
Friige
Dupzi
Figure 1: Demo results of the proposed ImageBrush framework various image manipulation tasks. By providing pair of task speciffccific examples and query image that share similar context, ImageBrush accurately identifies the underlying task and generates desired output.
Abstract
While language-guided image manipulation has made remarkable progress, the challenge of how to instruct the manipulation process faithfully reflecting human intentions persists. An accurate and comprehensive description manipulation task using natural language is laborious and sometimes even impossible, primarily due the inherent uncertainty and ambiguity present in linguistic expressions. it feasible to accomplish image manipulation without resorting to external cross- modal language information? If this possibility exists, the inherent modality gap would be effortlessly eliminated. In this paper, propose novel manipulation methodology, dubbed ImageBrush, that learns visual instructions more accurate image editing. Our key idea is to employ pair of transformation images as vixual instructions, which not only precisely captures human intention but also facilitates accessibility in real-world scenarios. Capturing visual instructions is particularly challenging because involves extracting the underlying intentions solely from visual demonstrations and then applying this operation to new image. To address this challenge, we formulate visual instruction learning as a diffusion- based inpaint-
"Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

ing problem, where the contextual information is fully exploited through an iterative process of generation. A visual prompting encoder carefully devised enhance the model's capacity in uncovering human intent behind the visual instructions. Extensive experiments show that our method generates engaging manipulation results conforming to the transformations entailed in demonstrations. Moreover, our model exhibits robust generalization capabilities on various downstream tasks such as pose transfer, image translation and video inpainting.
Introduction
Image manipulation has experienced a remarkable transformation in recent years [48, 14, 35, 71, 21, 29]. The pursuit of instructing manipulation process align with human intent has garnered significant attention. Language, as a fundamental element of human communication, is extensively studied to guide the manipulation towards intended direction [16, 34, 55, 33, 58, 5J. Despite the universal nature of ianguage-based instructions, there are cases where they fall short in expressing certain world concepts. This limitation necessitates additional efforts and experience in magic prompt engineering, as highlighted in [64]. To compensate linguistic ambiguity, some studies [65, 32, 66] attempt to introduce visual guidance the manipulation process. However, these approaches heavily rely on cross-modal alignment, which may not always be perfectly matched, thereby resulting in limited adaptability to user instructions.
Is it conceivable to accomplish image manipulation exclusively through visual instructions? If such a capability were attainable, it would not only mitigate the aforementioned cross- modallty disparity but also introduce a novel form of interaction for image manipulation. Taking inspiration from the exceptional in-context capabilities by large language models [37, 40, 7] like ChatGPT and GPT-4, we propose to conceptualize the image manipulation task as visual prompting process. This approach utilizes paired examples establish visual context, while employing the target image as a query. Recent works [61, 4] reveal the visual in-context ability through a simple Mask Image Modeling [15] scheme. But these researches focus on understanding tasks such as detection and segmentation while only very limited context tasks are studied. Therefore, the in-context potential for image manipulation, where numerous editing choices are involved, is still a promising avenue for exploration.
In contrast to language, which primarily communicates abstract concepts, exemplar-hased visual instruction explicitly concretizes manipulation operations within the visual domain. this way, the network could directly leverage their textures, which eases the hallucination difficulty for some inexpressible concepts (c.g., artist style or convoluted object appearance). On the other hand, the pairwise cases directly exhibit transformation relationship from the visual perspective, thereby facilitating better semantic correspondence learning. In this paper, we propose an Exemplar-Bast Based Image Manipulation framework, ImageBrush, which achieves adaptive image manipulation under the instruction of pair of exemplar demonstrations. This paradigm holds promising potential for various innovative applications. For instance, photographers seamlessly apply Photoshop-stye retouching to entire collections of similar images, enhancing their workflow. Additionally, users who encounter compelling editing instances from peers or pre-trained models can effortlessly apply these modifications to their own images, eliminating the need for original intricate intermediate steps or specific parameter adjustments.
The key is to devise a generative model that tackles both pairwise visual instruction understanding and image synthesis in an unified manner. To establish the intra-correlations among exemplar transformations and their inter-correlation with the query image, we utilize a grid-like image as model input that concatenates manipulation example and target query as [4, 61, 17). The in-context image manipulation is accomplished by inpainting the answer picture. Unlike their approaches forwarding once for final result, we adopt the diffusion process to iteratively enhance in-context learning and refine the synthesized image. Such practice not only provide stable training objectives [32], but also mimics the behavior of painter [9] who progressively fills and tweaks the details. Although the aforementioned formulation is capable of handling correlation modeling and image generation in a single stroke, places a significant computational burden on the network, due to the intricate concepts and processes involved. To address this challenge, we delicately design a visual prompting encoder that maximizes the utilization of contextual information, thereby alleviating the complexity of learning. Specifically, we extract the features of each image and further process them through transformer module for effective feature exchanging and integration. The obtained features are then injected to diffusion network with cross-attention to augment its understanding capability. In line

---

with the SAM concept [27], we have also introduced an optional interface design module aimed at further improving human intent comprehension.
Our main contributions can be summarized as follows: 1) We introduce novel image manipulation protocol that enables the accomplishment of numerous operations through an in-context approach. 2) We delicately devise a diffusion- based generative framework coupled with a hybrid context injection strategy to facilitate better correlation reasoning. 3) Extensive experiments demonstrate our approach generates compelling manipulation results aligned with human intent and exhibits robust generalization abilities on various downstream tasks, paving the way for future vision foundation models.
2 Related Work
Language-Guided Guided Image Manipulation. The domain of generating images from textual input [45, 47, 50] has experienced extraordinary advancements, primarily driven by the powerful architecture of diffusion models [19, 47, 51, 52]. Given rich generative priors in text-guided models, numerous studies [33, 11, 36, 2, 2] have proposed their adaptation for image manipulation tasks. To guide the editing process towards the intended direction, researchers have employed CLIP [44] to fine-tune diffusion models. Although these methods demonstrate impressive performance, they often require expensive fine-tuning procedures [24, 56, 25, 5]. Recent approaches [16, 34] have introduced cross- attention injection techniques to facilitate the editing of desired semantic regions spatial feature maps. Subsequent works further improved upon this technique by incorporating semantic loss [28] or constraining the plugged feature with attention loss [55].
Image Translation. Image translation aims to convert image from one domain to another while preserving domain-irellvant characteristics. Early studies [67, 30, 23, 60, 38] have employed conditional Generative Adversarial Networks (GANs) to ensure that the translated outputs conform to the distribution of the target domain. However, these approaches typically requires training a specific network for each translation task and relies on collections of instance images from both the source and target domains. Other approaches [1, 46, 53, 54, 57] have explored exploiting domain knowledge from pre-trained GANs leveraging augmentation techniques with single image to achieve image translation with limited data. Another slew of studies [70, 3, 59, 42, 22] focus on exemplar- based image translation due to their flexibility and superior image quality. While the majority of these approaches learn style transfer from a reference image, CoCosNet [72] proposes capturing fine structures from the exemplar image through dense correspondence learning.
In-Context Learning. In-Context Learning [7], concept originating from the realm of Natural Language Processing (NIP), offers an innovative approach to task completion. This paradigm achieves given task by providing it with set of sample examples alongside query example, demonstrating exceptional proficiency in executing these tasks through a few-shot learning mechanism. Expanding this notion, VisualPrompting [4] was pioneering endeavor that introduced the concept of visual in-context learning. This framework demonstrated its remarkable effectiveness in various applications, including image segmentation, object detection, and colorization, all implemented within the framework of Masked Image Modeling (MIM). Painter [61] further refined and extended the MIM process, broadening its scope to encompass tasks like key point detection and image denoising. Their subsequent work SegGPT [62] delved into the exploration of various intricate tasks within the domain of image segmentation. Notably, there are very few works investigating visual in-context learning from the perspective of image generation. Thus, this paper explores the possibility of in-context image manipulation, attempting to promote novel applications functioning as Iragh
3 Method
In this section, we will discuss the details of our proposed Exemplar-Based Image Manipulation Framework, ImageBrush. The primary objective of this framework is to develop a model capable of performing various image editing tasks by interpreting visual prompts as instructions. To achieve this, the model must possess the ability to comprehend the underlying human intent from contextual visual instructions and apply the desired operations to new image. The entire pipeline is depicted in Fig. 2, where we employ a diffusion-basee inpainting strategy to facilitate unified context learning and image synthesis. To further augment the model's reasoning capabilities, we meticulously design a visual prompt encoding module aimed at deciphering the human intent underlying the visual instruction.
3

---

Visual Prompt Design
Interface Design
Visual
Bbux
Bbox
Encoder
Cocrdinates
Eb
Bounding Box Drawing
Transformer
Grounding
"Tuming pomogranattes Fito lorelltiie
Prompt Encader
Transformer Lx
DINO
Langunge Description
Residual Block
Salf Attontion
Atfeation
F
Denoising U-Net Ep
Xt-1
X
Figure Illustration ImageBrush. We introduce novel and intuitive way of interacting with images. Users can easily manipulate images by providing a pair of examples and a query image as prompts to our system. users wish to convey more precise instructions, they have the option to inform the model about their areas of interest through manual bounding box annotations or by using language automatically generate them.
3.1 Exemplar-Based Image Manipulation
Problem Formulation. Given pair of manipulated examples {E, E'} and query image I. our training objective is to generate an edited image I' that adheres to the underlying instructions provided by the examples. Accomplishing this objective necessitates comprehensive understanding of the intra-relationships between the examples as well as their inter-correlations with the new image. Unlike text, images are not inherently sequential and contain multitude of semantic information dispersed across their spatial dimensions.
Therefore, propose solution in the form of "Progressive In In-Painting". A common approach is to utilize cross-ates tross-attention to incorporate the demonstration examples {E, E'} and query image I as general context like previous work [66]. However, observed that capturing detailed information among visual instructions with cross-attention injection is challenging. To overcome this, we propose concatenate learning their low-level context using self-attention mechanism. Specifically, blank image M to the visual instructions and compose grid-like image (E, E' I, M) as illustrated in Fig. 2. The objective is iteratively recover (E, I,',,')) from this grid-like image.
Preliminary on Diffusion Model. After an extensive examination of recent generative models, have determined that the diffusion model [47] aligns with our requirements. This model stands out due to its stable training objective and exceptional ability to generate high-quality images. operates by iteratively denoising Gaussian noise to produce the image zig. Typically, the diffusion model assumes a Markov process [43] wherein Gaussian noises are gradually added to clean image Fo based on the following equation:
1t ===oaz0 + VI- ate,
where ~ (O, represents the additive Gaussian noise, L denotes the time step and is scalar functions of f.. Our training objective to devise a neural network c0(x,, 1,c) to predict the added noise e. Empirically, simple mean-squared error is leveraged as the loss function:
Laimpee := ( ||c - c (zt, 2)
where 0 represents the learnable parameters of our diffusion model, and denotes the conditional input the model, which take the form of another image [49], class label [20], text [25].
By incorporating this condition, the classifier-fres guidance [45] adjusts the original predicted noise co 2) towards the guidance signal co(xı, c), formulated as
ég(I,, c) ca(x,, Ø) + wree(az, c) - cœ( Ø)).
(3)
The is the guidance scale, determining the degree to which the denoised image aligns with the provided condition

---

Context Learning by Progressive Denoising, The overall generation process is visualized in Fig.2. Given the denoised result -;-1 Grid E', from the previous time step t. our objective is to refine this grid-like image based on the contextual description provided in the visual instructions. Rather than directly operating in the pixel space, our model diffuses in the latent space of a pre-trained variational autoencoder, following similar protocol to Latent Diffusion Models (LDM)[47). This design choice reduces the computational resources required during inference and enhances the quality of image generation. Specifically, for an image Te, the diffusion process removes the added Gaussian noise from its encoded latent input 2 = E(r)). At the end of the diffusion process, the latent variable 20 is decoded obtain the final generated image 2e D(20). The encoder & and decoder 'D are adapted from Autoencoder- KL [47], and their weights are fixed in our implementation.
Unlike previous studies that rely on external semantic information [29, 6], here we focus on establish- ing spatial correspondence within image channels. We introduce a UNet-like network architecture, prominently composed of self-attention blocks, as illustrated in Fig. 2. This design enables our model to attentively process features within each channel and effectively capture their interdependencies.
3.2 Prompt Design for Visual In-Context Instruction Learning
However, relying only on universal correspondence modeling along the spatial channel may not be sufficient for comprehending abstract and complex concepts, which often require reassembling features from various aspects at multiple levels. To address this issue, we propose an additional prompt learning module enable the model to capture high-level semantics without compromising the synthesis process of the major UNet architecture.
Contextual Exploitation of Visual Prompt. Given the visual prompt vy - (E, E I], we aim to exploit their high-level semantic relationships. To achieve this, a visual prompt module comprising two components is carefully devised, which entails shared visual encoder e, and a prompt encoder ep as illustrated in Fig. 2. For an arbitrary image I E " W x3 within the visual prompt, we extract its tokenized feature representation fing using the Visual Transformer (ViT)-based backbone e,. These tokenized features are then fed into the bi-directional transformer ep. which effectively exploits the contextual information. The resulting feature representation fe encapsulates the high-level semantic changes and correlations among the examples.
Using the visual prompt, we can integrate high-level semantic information into the UNet architecture by employing the classifier-free guidance strategy as discussed in Section 3.1. This is achieved by injecting the contextual feature into specific layers of the main network through cross-attention.
8 = d'-1 + Conv(e'-1)
d- = 8' 11 SelfAtt(((')) 0 = 1 ++ CrossAttn(fe)
(5)
(6)
where o' denotes the input feature of the (-th block, and fe represents the context feature. Specifically, select the middle blocks of the UNet architecture, these blocks have been shown in recent studies to be responsible for processing high-level semantic information [58].
Interface Design for Enhancing User Intent Comprehension. Recent advancements in visual instruction systems have have showcased remarkable capabilities through the integration of human prompts [27]. Taking inspiration from this research, we introduce a similar interface module that empowers users to emphasize their areas of interest. This, in turn, enhances the system's 's understanding of user intention to some extent.
Users have the option to designate their area of interest either by manually drawing a bounding box or by utilizing automated tools [3]], as depicted in Fig. 2. Similar to [29], these selected boxes are first processed by box encoder, denoted as eb, before being integrated into the system. During the training phase, we employ DIONO [31] to label the focused region based on textual instruction. This approach not only alleviates the burden of manual labeling but also offers users the flexibility to opt for an automatic tool that leverages language descriptions to enhance their intentions, particularly in cases where drawing a bounding box is less preferable. Additionally, we randomly remove portion of the bounding boxes to ensure that our model fully harnesses the benefits of the visual in-context instructions.
5

---

Table I: Quantitative Comparison on In-the-wild Dataset.
Exemplar-
Ceemplar-Based Based Image Translation
Pose Transfer
Inpointing
Editing
Dataset
Scannett 12]
(R LRW(Mask)]10)
UBC-Fashion|69]
DAVIS1411
InstractPx22hx--ileeedd
Metric
FID4
FID.
FIDI
FID
SSIM1
FID
V-FIDI SSIM
SSIM
TSAM[73]
86.84
158.30
0.901
(oCosNed172)
19:49
15:44
14.25
38.61
0.885
VisusPPronp((4]
0.677
0.787
0.417
IrageBrush
9.18
9.67
8,95
12,99
0.910
18.70
175.25
0.816
0.437
4 Experiments
4.1 Experimental Settings
Datasets. Our work leverages four widely used in-the-wild video datasets Sccnnet (12), LRW [10), UBCFashion [69], and DAVIS [41]] as well as a synthetic dataset that involves numerous image editing operations. The Scannet [12] dataset is a large-scale collection of indoor scenes covering various indoor environments, such as apartments, offices, and hotels. It comprises over 1,500 scenes with rich annotations, of which 1, 1,201 scenes lie in training split, 312 scenes are in the validation set. No overlapping physical location has been seen during training. The LRW [10] dataset is designed for lip reading, containing over 1000 utterances of 500 different words with a duration of 1-second video. We adopt 80 percent of their test videos for training and 20 percent for evaluation. The UBC-Fashion [69] dataset comprises 600 videos covering diverse range of clothing categories. This dataset includes 500 videos in the training set and 100 videos in the testing set. No same individuals has been seen during training. The DAVIS [41] (Densely Annotated VIdeo Segmentation) dataset is a popular benchmark dataset for video object segmentation tasks. It comprises total of 150 videos, of which 90 are densely annotated for training and 60 for validation. Similar previous works [73], we trained our network on the 60 videos in the validation set and evaluate its performance on the original 90 training videos. The synthetic image manipulation dataset is created using image captions of LAION Improved Aesthetics 6.5+ through Stable Diffusion. We use the CLIP- filtered subset processed by InstruetPix2Pix [6]. It comprises over 310k editing instructions, each with its corresponding editing pairs. Out of these editing instructions, 260k have more than two editing pairs. We conduct experiment on this set, reserving 10k operations for model validation.
Implementation Details. In our approach, all input images have a size of 256x256 pixels and are concatenated as input to the UNet. The UNet architecture, adapted from Stable Diffusion, consists of 32 blocks with self-attention and cross-attention layers. We use cross-attention to incorporate the features of the visual prompt module into its two middle blocks. The visual prompt's ss shared backbone, denoted as e,, follows the architecture of EVA-02 [13]. The prompt encoder Ep comprises five layers and has latent dimension of 768. For the bounding box encoder, denoted as eg. we adopt simple MLP following [29]. During training, we set the classifier- free scale for the encoded instruction to 7.5 and the dropout ratio to 0.05. Our implementation utilizes PyTorch [39] and is trained on 24 Tesla V100-32G GPUs for 14K iterations using the Adam W [26] optimizer. The learning rate is set to le-6, and the batch size is set to 288.
Comparison Methods. To thoroughly evaluate the effectiveness of our approach, we compare it against other statc-of-art models tailored for specific tasks, including: VisualPrompt [4], the pioneer study in visual in-context learning: SDEdit [33]. widely-used stochastic differential equation (bse) image editing method; Instruct-Pixte Pix2pix [6], a cutting-edge language-instructed instructed image manipulation method; CoCosNet [72], approach that employs carefully designed correspondence learning architecture to achieve exemplar- based image translation; and TSAM [73], a model that incorporates a feature alignment module for video inpainting.
4.2 Quantitative Evaluation
Evaluation Metries. In the image manipulation task, it is crucial for the edited image align with the intended direction while preserving the instruction invirrian elements in their original form. To assess the degree of agreement between the edited image and the provided instructions, we utilize a cosine similarity metric referred to as CLIP Direction Similarity in the CLIP space. In order to measure the level of consistency with original image, we utilize the cosine similarity of the CLIP image embedding, CLIP Image Similarity, following the protocol presented in [6]. Additionally, for other generation tasks such as exemplar- based image translation, pose transfer, and video in in-painting,
we employ the Fréchet Inception Score (FID) [18] as an evaluation metric. The FID score allows

---

Comparison on Edit Direction
Comparison on Edit Similarity
0.25
1.00
Instruct-Po22ix ImageBrush
Instruct-Rx22pi
CLi Direction umilarity
ImageBrush
CLIP Similarity
0.95
0.15
0.90
0.10
0.85
0.05
0.80
0.80
0.85
0.90
0.95
1.00
0.00
0.05
10
0.15
0.20
0.25
mage Similarity
Direction Similarity
Figure 3: Quantitative Comparison on Image Editing. compare our approach with the representative text-instruction method in terms of direction consistency and similarity consistency.
us to evaluate the dissimilarity between the distributions of synthesized images and real images. Following similar protocol with [73], the V-FID also included to take into account the temporal consistency in video inpainting. For image generation, we introduce the SSIM [63] metric to further assess the structural similarity and visual quality.
Image Generation Paradigm. In the image manipulation task, a pair of transformed images is utilized as visual instructions to edit a target image within the same instruction context. For other tasks, conduct experiments in rross-frame prediction setting. where we select two temporally close frames and use one of them as an example to predict the other. Specifically, we employ one frame along with its corresponding label (e.g., edge, semantic segmentation, keypoints, or masked image) as a contextual example, and take the label from another frame as query to recover that frame. To obtain the label information, we extract keypoints from the UBC-Fashion dataset using OpenPose [8], and for the LRW dataset, we utilize face parsing network [68] to generate segmentation labels. To ensure that the selected frames belong the same context, we restrict the optical flow between them to specific range, maintaining consistency for both training and testing. It is important to note that, for fair comparison, we always utilize the first frame of each video as reference during the evaluation of the video inpainting task.
Evaluation Results. The comparison results for image manipulation are presented in Fig. 3. We observe that results exhibit higher image consistency for the same directional similarity values and higher directional similarity values for the same image similarity value. One possible reason is the ability of visual instructions express certain concepts without modality gap, which allows method to better align the edited images.
To demonstrate the versatility of our model, we conducted experiments using in-the-wild videos that encompassed diverse real-world contexts. The results for various downstream tasks are presented in Table 1, showing the superior performance of our approach across most datasets. It is noteworthy that our model achieves these results using a single model, distinguishing from other methods that require task-specific networks. In inpainting task, our model attains higher FID score while yielding comparatively lower Video-FII score and SSIM value. We speculate that it because our framework lack explicit temporal awareness such as optical flow in [73].
4.3 Qualitative Evaluation
We provide qualitative comparison with SDEdit [33] and Instruct-Pix2pix [6] in Fig. 4. In the of the SDEdit model, we attempted to input both the edit instruction and the output caption, referred as SDEdit-E and SDEdit-OC. respectively. In contrast language-guided approaches, our method demonstrates superior fidelity to the provided examples, particularly in capturing text-ambiguous concepts such as holistic style, editing extent, and intricate local object details. Comparison with VisualPrompt [4] is depicted in Fig. 6 where we achieve more realistic and plausible editing results.
Additionally, we provide qualitative analysis on real-world dataset in Fig. 5. Compared to CocosNet [72], our approach demonstrates superior visual quality preserving object structures, as seen in the cabinet and table. It also exhibits higher consistency with the reference appearance, particularly noticeable the woman's hair and dress. Furthermore, our method achieves improved shape consistency with the query image, observed in the mouth. When compared to TSAM [73], our

---

Edit Instruction
Query Image
SDEOit (E)
SDEdkt (OC)
Instruct-Pix2pix
Ours
The picture of street in Paris
The tiger cub snow leogard
Tum snowy landscape
Tum the bridge into bridge made of ice
Figure Qualitative Comparison on Image Manipulation. contrast to language- guided approaches, our editing results guided by visual instruction exhibit better compliance with the provided examples.
Scannet
UBC-Fashion
LRW
DAVIS
Query
Instruction
GT
Others
Ours
Figure 5: Qualitative Comparison on In-the-wild Dataset. We conduct experiments on various downstream tasks including exemplar-based image translation, pose transfer and inpainting.
approach yields superior results with enhanced clarity object shapes and more realistic background filling. These improvements achieved by effectively leveraging visual cues from the examples.
4.4 Further Analysis
Novel Evaluation Metric. Exemplar- image manipulation a novel task, and the evaluation metric in the literature is not suitable for this task. Therefore, present evaluation metric

---

ViesslPrompp
ImageBrnsh
ViuulPrempp
Imageitrash
VisusPPrempi
ImageRBred
Figure 6: Comparison with VisualPrompt [4] on Image Editing. Our model achieves plausible and realistic edits whereas VisualPrompt tend to produce conservative prediction and struggle on instruction comprchension.
0,1
Figure 7: Case Study in Terms of Prompt Fidelity and Image Fidelity. The prompt fidelity and image fidelity calculated between paired images with dashed yellow banner and dashed blue banner, respectively.
that requires neither ground-truth image nor human evaluation. Specifically, our model, denoted Qu. takes the examples { E'J. and the source image produce manipulated output I' This procedure is presented as I' &a E - E). The goal is to let the output image I' abide by instruction induced from examples E and E' while preserving the instruction-invariant content of input image I. We define prompt fidelity to assess the model's ability to follow the instruction. According the symmetry of E, E'} and {I, ['). if its operation strictly follows the instruction, the model should manipulate E to E' by using I + I' as the prompt (see pictures labeled with a dashed yellow tag in Fig. 7). We express the prompt fidelity as follows
Aprompt = FID(E', 2 + I))).
(7)
On the other hand, to evaluate the extent to which the model preserves its content, we introduce an image fidelity metric. If I' maintains the content of I, the manipulation should be invertible. That is to say, by using E' > E as the prompt, the model should reconstruct from I' (see pictures labeled with dashed blue tag in Fig. 7). Therefore, define the image fidelity
Ammage FID & (I' E)).
(8)
Ablation Study. We performed ablation studies on three crucial components of our method,
namely the diffusion process for context exploitation, the vision prompt design, and the injection of iumman-interest area. Specifically, we conducted experiments on our model by (1) replacing the diffusion process with masked image modeling, (2) removing the cross-attention feature injection obtained from vision prompt module, and (3) deactivating the input of the human interest region. Specifically, implemented the masked image modeling using the MAE-VQGAN architecture following the best setting of [4]. The numerical results on image manipulation task are shown in Table 2. The results demonstrate the importance of the diffusion process. Without it, the model is unable to progressively comprehend the visual instruction and refine its synthesis, resulting in inferior generation results. When we exclude the feature integration from the visual prompt module, the model struggles to understand high-level semantics, leading to trivial generation results on these instructions. Furthermore, removing the region of interest interface results in a slight decrease in performance, highlighting its effectiveness in capturing human intent.
Case Study. Fig. 7, we present two cases that showease our model's performance terms of image and prompt fidelity. The first case, depicted the left, highlights our model's capability to

---

Table 2: Ablation Study on Image Manipnlation. Here evaluate proposed metric.
w/o Diffusion
w/o Aross-Attention
Interest Region
Full Model
Prompt Fidelity Image Fidelity 82.33
78.65
39.44
24.29
23,97
41.51
25.74
24.43
Figure 8: Generation Results by Customized Prompts. The regions of interest are indicated by purple box and generated results are displayed within red box. From left right illustrate cases where emphasis is placed on beards, hats, buckets, flying objects and trees.
Figure 9: Image Editing with Varied Queries. We showcase edited results corresponding to three distinct queries. covers a variety query contexts, demonstrating changes layout, object quantity, weather, pose, shape and perspective.
utilize its predicted result to reconstruct I by transforming the image back to a dark appearance. With predicted I', our model also successfully edits the corresponding example from E to E', making the example image colorful. On the right side of the figure, we present a case where our model encounters difficulty in capturing the change in the background, specifically the disappearance of the holographic element.
Effeet of User Interface. Fig. 8 depicts diverse examples with distinct emphasis on visual instruc- tions. For example, synthesis of additional beards when the beard area is emphasized, the generation of more detailed hats when a bounding box is applied to the hat part, and the improved ability of our model to understand and execute the intention of editing a bucket black when the emphasis is on the bucket. Furthermore, when small object is added, our model successfully comprehends and synthesizes an object with the assistance of this input. Similarly, by placing emphasis on the background, more trees are synthesized in that region.
Image Editing with Diverse Queries. We present collection of qualitative examples encom- passing diverse query contexts in Fig. 9. The results suggest our model's competence in generating coherent and reliable outcomes across various query scenarios.
5 Conclusion
In this article, present innovative way of interacting with images, Image Manipulation by Visual Instruction. Within this paradigm, we propose a framework, ImageBrush, which holds promising potential for various applications.
Limitation and Future Work. 1) The proposed model may face challenges when there is substantial disparity between the given instruction and the query image. 2) The current model also encounters difficulties in handling intricate details, such as subtle background changes or adding small objects. 3) Future work can delve into a broader range of datasets and tasks, paving the way for future generative visual foundation models.
10

---

References
[1] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: residual- stylegan encoder via iterative refinement, Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2021.
[2] Omri Avrahami, Ohad Fried, and Dani Lischinski. urXiv:2206.027777 02779, 2022.
Blended latent diffusion. arXiv preprint
[3] Aayush Bansal, Yaser Sheikh, and Deva Ramanan. Shapes and context: In-the-wild image synthesis & manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2317 2326, 2019.
[4] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Gloherson, and Alexei Efros. Visual prompting via image inpainting. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems, volume 35. pp. 25005-25017, Curran As- sociates, Inc., 2022. URL https::/ http://///poccco..../oo iewpreeeeeiinnnnnniiiieeeeeeeeee neurips. cc/ cc/paper__. wttp://eeesseaaee222//////// Paper-- Conference. pdf.
[5] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Computer Vision-ECCH, 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 707-723. Springer, 2022.
[6] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. 2020.
Advances in neural information processing systems,
[8] Zhe Cao, Tomas Simon, Shih-En En Wei, and Yaser Sheikh. Realtime multi-person part affinity fields. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 7291-7299, 2017.
person 2d pose estimation using
[9] Huiwen Chang, Han Zhang, Lu Jiang. Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
[10] J. S. Chung and A. Zisserman. Lip reading in the wild. In Asian Conference Computer Vision, 2016.
[11] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthicu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv preprint ar arXiv:2210.11227, 2022.
[12] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias NieBner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
[13] Yuxin Fang. Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303. 10000///133/ 11311, 2023.
[14] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion, 2022. URL https://arxiis org/ a2wg/088./2222.... 01618.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 16000- 16009, June 2022.
[16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint ar X2v: 01626, 2022.
[17] Aaron Hertzmann, Charles E Jacobs, Nuria Oliver. Brian Curless, and David H Salesin. Image analogies. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp. 327-340, 2001.
11

---

[18] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. Von Luxburg. Advances Neural Information
S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett Processing Systems, volume 30, Curran Associates, Inc., 2017. URL https: / httpp//oooo....eeeeuue neurips. co/paper pallee//aper//000/ httpppppr0011111/a/a////00000000 / 157.e Paper pdf.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems. 33:6840-6851 2020.
[20] Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23 (47):1-33, 2022. URL http: / jmlr wtggpaperss//3//22
0635. html.
[21] Lianghua Huang, Di Chen, Yu Liu, Shen Yujun, Deli Zhao, and Zhou Jingren. Composer: Creative and controllable image synthesis with composable conditions. 2023.
[22] Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV). September 2018.
[23] P. Isola, Zhu, T. Zhou, and networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967- 5976, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society. doi: 10. 10.1109/C2.. URL https: //doi. iesecompttersooiettyy org/ 10. 1109/CVPR.
Efros. Image-to-image translation with conditional adversarial
[24] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Trani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276. 2022.
[25] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2426 2435,22435222 2022.
[26] Diederik P. Kingma and Jimmy Ba, Adam: A method for stochastic optimization. CoRR, abs/1412,6980, 2014.
[27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. mnXiv:2.04.026644 2023.
[28] Gihyun Kwon and Jong Chul Ye. Diffusion--asee image translation using disentangled style and content representation. arXiv preprint arXiv:2209. 220..22409,,44,, 2022.
[29] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. CVPR, 2023.
[30] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to -oe--ttoomeeee-eeee translation net- works. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: / /ttpc//eeoccceeeeeeeeeeeeece/.ee meurips.co httpecpaepppeplleepeeaaeeee wwt.s///pppaee/11/////// 010a6669944002200442222666666666 Pappeeeerbbbb--aaa....eeree pdf.
[31] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwci Yang, Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint MrXiv:2303.05499, 2023.
[32] Yiyang Ma. Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu. Unified modal latent diffusson for joint subject and text conditional image generation. arXiv preprint arXiv:2303.0m 09319, 2023.
[33] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. International Conference on Learning Representations, 2022.
[34] Ron Mokady, Amir Hertz, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:221 1. 1.09994, 2022.
[35] Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to to-image diffusion models. arXiv preprint arXiv:2302.0845 08453, 2023.
12

---

[36] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided difffusion models. arXiv preprint 112.v22112.0741, 2021.
[37]
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Curran Associates, Inc., 2022. URL https: http::pppp//ppoooeednnneeee wtwp://ceeeiiinnnnninieieeeeeeee cc/ cc/paper. filles/paper22022// wtttee//eeeeeeeeeeeeeeeeeee 01146660779999119960000000111113 Paper----ppp--aae Conference . pdf.
[38] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu, Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019.
[40] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:234.002277 2023.
[41] F. Perazzi, Pont-Tuset, B. McWilliams, I.. Van Gool, M. Gross, and A. Borkine-Homung A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Pattern Recognition, 2016.
[42] Xiaojuan Qi, Qifeng Chen, Jiaya Jia, and Vladlen Koltun. Semi-parametrie image synthesis. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
[43] Lawrence R Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2)2577-266 1989.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
[45] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[46] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: A stylegan encoder for image-lo-imad translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2287-2296, June 2021.
[47] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusin models. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
[48] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[49] Chitwan Saharia. William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-iman diffiusion models. In ACM SIGGRAPH 2022 Confer- ence Proceedings, SIGGRAPHI '22, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393379. doi: 10. 00011455/5222333533772 URL http:///s////o/ org/ 10. 1145/3528233, 3530757.
[50] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan. Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, [51] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint arXiv:2010.02502, 2020.
2022.
13

---

[52] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[53] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Trans. Gruph. 40(4), jul 2021. ISSN 0730-0301. doi: 10.1145) 3450626.3459838. URL https:/ /doii org/10. 1145/3450626. 3459838.
[54] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2022.
[55] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text- driven toaggeto-image translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[56] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image editing by fine tuning an image generation model on single image. arXiv preprint arXiv:2210.09477 2022.
[57] Yael Vinker, Eliahu Horwitz, Nir Zabari, and Yedid Hoshen. Image shape manipulation from a single augmented training sample. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13769-13778, 2021.
[58] Andrey Voynov, Qinghao Chu, Daniel Cohen- Or. and Kfir Aberman. p+: Extended textual conditioning text-to-image generation. arXiv preprint arXiv:2303.099 09552, 2023.
[59] Miao Wang, Guo-Ye Yang, Ruilong Li, Run-Ze Liang, Song-Hai Zhang, Peter M Hall, and Shi-Min Hu. sxample-guided cylessconisttet image synthesis from semantic labeling. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1495-1504, 2019.
[60] Ting-Chun Wang, Ming-Y Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8798 8807, 2017.
[61] Xinlong Wang. Wen Wang, Yue Cao. Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[62] Xinlong Wang. Xiaosong Zhang, Yue Cao. Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything context. ArXiv, abs/2304.03284, 2023. URL org/CorpusID : 557998532.
[63] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processiing 13(4);600-6/) 612, 2004.
[64] Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-bard based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.0366 2023.
[65] Eric Zhang Kai Wang Humphrey Shi Xinggian Xu, Zhangyang Wang. Versatile diffusion: Text, images and variations all in diffusion model. 2022. URL https://arxiv.. org. abs /2111. 08332..
[66] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffuusio models. arXiv preprint arXiv:2211.1322 2022.
[67] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2868-2876. 2017. doi: 10.1 10//01/.22222....111..
[68] Changqian Yu, Changxin Gao, Jingbo Wang. Gang Yu, Chunhua Shen, and Nong Sang. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. Int. J. Comput. Vision, 129(11):051-3366 nov 2021. ISSN 0920-5691. doi: 10.10077111225-001-001554 //doi 1iloiioorg/0..1000 12210/111122-220011555
URL https:
[69] Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for guided human video generation. In British Machine Vision Conference, 2019.
[70] Bo Zhang, Mingming He, Jing Liao, Pedro Sander, Lu Yuan, Amine Bermak, and Dong Chen. Deep exemplar- based video colorization. In Proceedings of the IEEE/CVF conferncco on computer vision and pattern recognition, pp. 8052 000-8061, 2019.
14

---

[71] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arX1v:2302.05543, 2023.
[72] Pan Zhang. Bo Zhang. Dong Chen, Lu Yuan, and Fang Wen. Cross-domain correxpondence learning for exemplat-based based image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5143-5153, 2020.
[73] Xueyan Zou, Linjie Yang. Ding Liu, and Yong Jae Lee. Progressive temporal feature alignment network video inpainting. In CVPR, 2021.
15

---

Appendices
A Analysis of In-Context Instruction with Multiple Examples
Our approach can be naturally extended to include multiple examples. Specifically, given a series of examples [E1, Ei...., En, E, IJ, where represents the number of support examples, our objective is to generate I' In our main paper, primarily focused the special case where n 1. When dealing with multiple examples, we could also establish their spatial correspondence by directly concatenating them as input to our UNet architecture. Specifically, we create a grid 2;-1 Grid( E1, Ei...., En, E, x 1) that can accommodate up to eight examples following [4]. To facilitate contextual learning, we extend the input length of prompt encoder ep. and incorporate tokenized representations of these collections of examples as input to it. In this way, framework is able to handle cases that involve multiple examples.
Below discuss the impact of these examples on our model's s final performance by varying their numbers and orders.
Prompt-Fidellty Image- Pidee-Fidlltyy
Synthesis Fidelity (FDD)
2
Number of Support Examples
Instruction with Support Examples
Figure 10: Analysis of In-Context Instruction with Multiple Examples.
A.1 Number of In-Context Examples.
our dataset, which typically consists 4 examples, we examine the influence the number in-context examples by varying from to 3, as illustrated in Figure 10. We evaluate this variation using our proposed metries: prompt fidelity Apromp and image fidelity Amage. which assess instruction adherence and content consistency, respectively. The results demonstrate that inereasing the number of support examples enhances performance, potentially by reducing
ambiguity.
Furthermore, we present additional instruction results in Figure 11, showcasing the impact of different numbers in-context examples. is clear that as number of examples increases, the model becomes more confident in comprehending the task description. This is particularly evident in the second row, where the synthesized floor becomes smoother and the rainbow appears more distinct. Similarly, in the third row, the wormhole becomes complete. However, for simpler tasks that can be adequately induced with just example (as demonstrated in the last row), increasing the number of examples does not lead further performance improvements.
A.2 Order of In-Context Examples.
We have also conducted an investigation into impact of the order support examples by shuffling them. Through empirical analysis, we found no evidence suggesting that the order of the examples has any impact on performance. This finding aligns with our initial expectation. as there is no sequential logic present in our examples.
B More Visual Instruction Results
this section, we present additional visual instruction results, where synthesized result is highlighted within the red box. These results demonstrate the effectiveness of our approach in handling diverse manipulation types. including style transfer. object swapping. and composite operations. as showcased in Fig. 12 and Fig. 13. Furthermore, method demonstrates its versatility downstream tasks, such as image translation, pose translation, and inpainting in Fig. 14 and Fig. 15. Importantly, is worth noting that the presented results are generated by single model.
real-world datasets, successfully tackling various
16

---

HP
199
1
Figure 11: Qualitative Analysis on Number of In-Context Examples.
17

---

Figure 12: Image Manipulation Results by Visual Instruction.
18

---

Figure 13: Image Manipulation Results by Visual Instruction.
19

---

Figure 14: Instruction Results on In-the-wild Dataset.
20

---

Figure 15: Instruction Results on In-the-wild Dataset.
21


Reviews:
[{"review_id": "SRf3GZGiV2", "reviewer": "Reviewer_RQEf", "summary": "The paper proposes a novel image manipulation technique based on diffusion and visual context images. The authors argue that context from language can only communicate abstract ideas while giving visual context can bring across inexpressible concepts like art style or complex appearances.\n\nThe novel technique adopts the 4-image-quadrant concept where the 4th quadrant is in-painted by to model to match some target (similar to SegGPT and Painter). But instead of in-painting segmentation masks, depth maps, or key-points in the 4th quadrant, they in-paint an edited version of the query based on transformation style or content given the context (prompts).\n\nTo improve the grounding process, this work proposes to add an additional prompt encoder branch of the architecture which is fed into the diffusion process via Cross-Attention. This branch also allows optional guidance through bounding boxes on which the in-painting process should add additional focus. Bounding boxes can either be provided by drawing them or via language by using Grounding Dino.\n\nFinally, this work introduces a novel evaluation metric for exemplar-based image manipulation which does not need ground truth or human evaluation.", "questions": "Would the architecture still work if you would drop the 4-quadrant concept for two quadrants and only injecting context/prompts through cross attention?", "limitations": "Authors adequately address limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-written and mostly easy to read.\n- The proposed approach is novel and the paper does a good job of highlighting the motivation for this line of work. The method is evaluated on multiple benchmarks and shows promising results. Comparisons are made with other image manipulation methods wherever possible.\n- The work is visually attractive with many examples of generated images and comparisons with similar methods\n- The authors highlight limitations of their approach by showcasing examples where the model works well but also cases where the model breaks down and does not work well.\n- Authors show qualitative and quantitative results on multiple datasets, showcasing the superiority of their method.", "weaknesses": "- The description of the design of the prompt encoder is minimal and it would be interesting to have a more in-depth description. Specifically to how the different context image embeddings are combined. Are they just concatenated like the bounding box embeddings?\n- The section on the novel evaluation metric is somewhat unclear, specifically why the two directions of evaluation/combination of query and prompts are chosen. (Maybe more descriptive variable naming could help)\n- Generally, captions of figures and tables are minimal and contain little information. Specifically, figure 6 is confusing and could use an explanatory caption.\n- Equations 4 to 6 are unclear and not understandable.\n- Supplementary material is not formatted as an appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\n**_How the different context image embeddings are combined._**\n\nWe apologize for the lack of clarity in the description of the prompt encoder and will refine this section for improved clarity. Yes, the different context image embeddings are indeed concatenated, just like the bounding box embeddings. Throughout our investigation, we explored alternative strategies, such as leveraging differences in CLIP features between instruction images for information injection, as well as incorporating supplementary operations to characterize editing procedures. However, our experimentation indicated that these approaches failed to yield substantial advantages. It turns out the uncomplicated concatenation of embeddings already demonstrates efficacy in facilitating the model's comprehension of the editing intent derived from multiple images. Hence, we opted for straightforward concatenation for the sake of simplicity.\n\n\n\n**_The novel evaluation metric is somewhat unclear._**\n\nWe apologize for any confusion that may have arisen in our explanation of the novel evaluation metric. We will revise those parts to provide greater clarity and incorporate descriptive variable names. The rationale behind the choice of the two directions of evaluation and the combination of query and prompts is to assess the fidelity of our model's image manipulation in terms of both following the provided instructions (prompt fidelity) and preserving the content of the input image (image fidelity).\n\n* Prompt Fidelity: This metric evaluates the model's ability to follow the provided instructions, which are given in the form of exemplar image pairs. By using the instructions formulated from the query image and generated image, we expect the model to manipulate one of the example images to match the other one. A high prompt fidelity score indicates that the model effectively captures the underlying instructions from the examples and successfully applies the desired transformation.\n\n* Image Fidelity: This metric evaluates the extent to which the model preserves the content of the input image when performing the manipulation. If the manipulated image maintains the content of the input image, the manipulation should be somehow invertible. By using the reverse instructions from the examples, we expect the model to reconstruct the original input image from the manipulated one. A high image fidelity score indicates that the model successfully preserves the instruction-invariant content of the input image during the manipulation process.\n\nIn summary, the combination of query and prompts in the evaluation metric aims to assess the model's performance in capturing the underlying human intent from the visual instructions and applying the desired operations to a new image, while preserving the instruction-invariant content of the input image. We believe that this combination of directions provides a comprehensive evaluation of our model's capabilities in the context of exemplar-based image manipulation tasks. We will clarify this part in the manuscript and tune down its claim.\n\n**_Equations 4 to 6 are unclear._**\n\nWe apologize for the lack of clarity in Equations 4 to 6. This section introduces the \"Residual Block - Self Attention - Cross Attention\" structure in diffusion model's UNet denoise predictor. In the original presentation, we used an excessive number of overlapping notations, making the description vague and unclear. We will rewrite this part to provide a clearer explanation. Thank you for your valuable feedback.\n\n**_Supplementary material is not formatted as an appendix._**\n\nWe apologize for the formatting issue in the supplementary material. We will ensure that the supplementary material is properly formatted as an appendix in the revised version of the paper. \n\n**_Only injecting context/prompts through cross attention?_**\n\nIt will not work effectively, because relying solely on cross-attention makes it challenging to preserve low-level information, which in turn reduces the quality of image editing. Low-level information is crucial, as many tasks require direct reference from the query image $\\textbf{I}$ or instruction images $\\textbf{E}$ and $\\textbf{E}'$, such as replicating the unique shapes drawn by users in the instructions.\n\nOur primary motivation for constructing the 4-quadrant inpainting task is to inject low-level features from $\\textbf{E}$, $\\textbf{E}'$, and $\\textbf{I}$ through the self-attention branch rather than relying solely on cross-attention. If we do not construct this inpainting task and solely incorporate features into the UNet's cross-attention to control diffusion, the image features would lose a significant amount of detail after being encoded by CLIP. Moreover, since the injection takes place in the latent space of LDMs, low-level information must be restored using the knowledge of the VAE's encoder.\n\nBy using the grid-like image inpainting approach, features of $\\textbf{E}$, $\\textbf{E}'$, and $\\textbf{I}$ are fully aligned with the predicted image $\\textbf{I}'$, as they all pass through the VAE's encoder. This method allows us to retain as much low-level information as possible in a simple and effective manner. If the loss of resolution is a concern, we can adopt a channel concatenation approach (however, this would require allocating separate parameters for each image to avoid alignment issues).\n"}, {"review_id": "d3daWKNPI3", "reviewer": "Reviewer_zhtQ", "summary": "In this paper, the authors propose a new method that can perform visual prompting via a pair of exemplar images. The authors introduce and train a new model for this task and evaluate the model performance on various datasets, showing strong empirical performance and a wide range of applications. They also propose new evaluation metrics to evaluate their method.", "questions": "1. How general is ImageBrush? How does the model perform on more OOD tasks/inputs (i.e. the prompts to which similar examples are not seen in training)? Or does it require training for each task?\n2. How similar does the visual prompt and query image should look? Can the authors give some additional ablation study on the similarity between the visual prompt and thequery images?\n3. Related to Weakness (1), can the authors compare their method with Visual Prompting via Image Inpainting?\n4. Related to Weakness (2) and (3), can the authors report some standardized metrics for tasks like semantic segmentation mask to image, keypoint to image and user-specified regional editing?\n\n~~I am happy to raise my score if the authors add experiments to address all my concerns.~~\n\n***After the rebuttal discussion, I would like to raise my score from 5o to 7.***", "limitations": "Although presented in the paper, the discussion of limitations and potential ethical issues is very limited. I would encourage the authors to extend this discussion by providing some failure cases and suggesting some future plans for safeguards, etc.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This model introduced in this paper solves a very interesting and important application problem that is exemplar-based visual prompting of image generative models. The results look very promising and this model can be easily directly applied in a large range of intuitive applications by daily users.", "weaknesses": "1. This paper is very similar to the paper “Visual Prompting via Image Inpainting” (https://arxiv.org/pdf/2209.00647.pdf). However, the authors didn’t compare their method with this paper.\n2. The quantitative evaluations are not very comprehensive. For example, for tasks like semantic segmentation masks to image, the authors could have reported some standardized metrics like IoU.\n3. Similar to (2), quantitative analysis on user-specified regional editing can also be included by comparing visual distances such as LPIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\n**_Weakness_**\n\n1. We further include additional comparison results against VisualPrompt in our appended PDF document. Fig.4 illustrates the qualitative results, revealing that ImageBrush yields plausible and realistic edits whereas VisualPrompt tends to produce conservative predictions and struggles on instruction comprehension. We believe this advantage can be attributed to the progressive synthesis nature of Diffusion. Moreover, quantitative results in Table.1 demonstrate the superiority of our approach against VisualPrompt on most tasks.\n\n2. The appended PDF document also includes a quantitative evaluation with standard metrics, such as PSNR, SSIM, and LPIPS. As depicted in Table 1, our approach demonstrates superior performance in terms of SSIM and LPIPS scores, highlighting its effectiveness in maintaining perceptual quality and preserving high-level structures.\n\n3. Regarding the quantitative analysis of user-specified regional editing, we have also included the corresponding results in Table 1. Notably, the \"ImageBrush-w-BB\" model takes the region of interest (ROI) into consideration. However, we observe that there is no significant divergence in terms of LPIPS, PSNR, and SSIM metrics. This outcome could be attributed to the inherent diversity of the editing process, which might not always align precisely with a specific ground truth (GT). As a result, we have conducted an additional evaluation using CLIP direction similarity, revealing superior performance when enhanced with user-specified region injection (0.18 vs. 0.17). Furthermore, we incorporate more qualitative results to further demonstrate the effectiveness of this module in Fig. 1. \n\n**_Questions_**\n\n1. To assess the model's generalizability, we have included extra in-the-wild examples in the PDF file. In the center of Figure 3, we showcase edited outcomes using real-world query images sourced from the Internet. Despite the distinct nature of these query images compared to the training distribution and example instructions, our model is capable of generating plausible results. On the right-hand side, we present out-of-distribution (OOD) cases. Notably, our model was not exposed to animal edges and keypoints, depth maps, or facial normals during the training phase. However, it demonstrates the ability to grasp the primary intent and produce reasonable edits. These examples highlight the model's capacity to learn fundamental visual correlations even in previously unseen scenarios to a certain extent.\n\n2. Regarding the alignment between visual prompts and query images, we present a collection of qualitative examples encompassing diverse query contexts in the attached document (Fig. 2). The results suggest our model's competence in generating coherent and reliable outcomes across various query scenarios. Generally, a query image with a CLIP Similarity surpassing 0.7 tends to yield satisfactory performance. To analyze the relationship between CLIP Image Similarity and CLIP Direction Similarity, we calculate a correlation matrix and obtain a Pearson Correlation Coefficient of 0.195, accompanied by a T-Statistic of 10.58, implying that higher similarity leads to better performance to a certain extent. \nFurthermore, we present an additional ablation study concerning their relationship listed below. From the results we can see that, on the whole, editing performance demonstrates improvement as the query image aligns more closely with the example instructions.\n\n\n| Ablation Study|  |  |  |  |  |  |  |  |  \n|---------------------------|------|------|------|------|------|------|------|------|  \n| CLIP Image Similarity     | 0.76 | 0.79 | 0.82 | 0.85 | 0.88 | 0.91 | 0.94 | 0.97 |  \n| CLIP Direction Similarity | 0.10 | 0.12 | 0.12 | 0.13 | 0.15 | 0.17 | 0.21 | 0.27 |  \n\n\n\n3. Addressed in Weakness 1.\n\n4. Addressed in Weakness 2.\n\n\n**_Limitation_**\n\nThank you for the valuable feedback. In our future revisions, we will certainly expand the limitations section as suggested. This expansion will encompass the visualization and analysis of failure cases. In our empirical observations, we've noted challenges when modifying small objects, possibly stemming from confusion with their semantic concept. \nAs for the future work, our framework favors spatially aligned instruction examples because most  instances within our dataset are aligned. Therefore, our future endeavors will focus on integrating language cues to improve the comprehension of small objects, while also exploring visual instruction with non spatially aligned example images.\n\n"}, {"review_id": "UaG4FmGtfy", "reviewer": "Reviewer_h3g5", "summary": "This paper introduces a novel image manipulation framework called ImageBrush, which allows for accurate and efficient image manipulation without the need for complex language descriptions. The framework consists of a diffusion-based generative model coupled with a hybrid context injection strategy, which facilitates better correlation reasoning. The authors also introduce a bounding box incorporation module to take user interactivity into consideration. The framework is evaluated on various downstream tasks and exhibits robust generalization abilities. Overall, the results demonstrate that ImageBrush generates compelling manipulation results aligned with human intent and paves the way for future vision foundation models.", "questions": "N/A", "limitations": "See weaknesses.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper introduces a new image manipulation framework called ImageBrush, that does not rely on complex language descriptions. This may help mitigate severe prompt engineering efforts in applications.\n-  A diffusion-based generative model coupled with a hybrid context injection strategy is proposed, which is a technically rigorous approach to image manipulation.\n- Extensive experiments are conducted on various downstream tasks and discuss this work's generalization abilities.\n- The paper is well-written and easy to follow, with clear explanations of the technical concepts and experimental results.", "weaknesses": "- Although this work utilizes an in-context learning strategy to avoid heavy prompt engineering effort, the image prompt itself seems to arise another challenge. Take Fig. 4 as an example, the provided instruction (tiger pairs) itself necessitates heavy editing skills, that require another expert (either human or modern algorithms) to provide a snowy tiger. I'm worrying that this will not be practical in real scenarios given users mostly can not provide such a good pair of instructions, and writing prompts would be relatively easier. A more practical way is to provide a tiger with another snowy example (not have to be a tiger) so that it will be easier to collect inputs. However, this paper does not show this demo.\n\n- Besides the input collecting issue, it seems that the instruction and the input image are very similar in all provided results. If that is the requirement of making the proposed algorithm succeed, it will be further difficult to be applied to real scenarios, since one can not pre-collect corresponding instructions due to the in-the-wild testing case is not predictable.\n\n- The authors propose a new evaluation metric in Sec 4.4. I agree that instruction-based editing needs a reliable quantitative metric to help justify the performance of each algorithm, but I doubt whether it is a good idea to put it as one of the contributions. Proposing a new metric normally requires a large amount of user study that can faithfully provide evidence to show that the new metric is aligned with human perception, which is lacking in this work. I understand that will cost a huge effort and would probably produce a completely new submission. Therefore, I would suggest the author not claim this as a \"novel\" finding and lower its importance in the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\n**_Visual instruction collection seems a challenge._**\n\nAs stated in the common response, we highlight that the practical application of our proposed system is to offer an efficient tool for users to _automatically streamline their desired editing operation to a new-coming image_, which acts like the Format Painter in Office Software, rather than _targeting editing a specific image_. Specifically, it is natural and intuitive to instruct an image with language considering its easy access. But when it comes to a situation where users want to apply analogous manipulation referring to his or others’ finished edits. In this case, it is particularly challenging to tackle with text condition or single image conditioned diffusion models because it requires inducing the underlying operations and imposing the intended manipulation on the target image. Furthermore, since the proposed system behaves like a ImageBrush, users are capable of accomplishing this solely with a pair of images without recording tedious settings such as a series of parameters in Photoshop. \n\nTo sum it up, users could always manually create or find their preferred visual instructions, with which our model enables them to conveniently leverage the implicit operations. Admittedly speaking, we agree that this framework will be more user-friendly if it could support visual instructions without relying on spatial alignment of the example image. At the same time, it’s quite an intriguing idea to obtain a snowy tiger when prompting for tiger and snow, which shares a similar setting with CoDI. However, this type of prompting is a little bit different from our work where we rely on paired manipulated images to indicate the desired editing operation. Regarding how to leverage these soft visual prompts, we will leave it for future work.\n\n\n**_Similarity in instruction and query image constraints real-world scenario applications._**\n\nAs depicted in the accompanying PDF document, Figure 2 demonstrates our method's versatility in image editing across a diverse range of query inputs, which suggests our model's capability to handle a wide spectrum of query scenarios. Additionally, Figure 3 serves to illustrate our framework's adaptability – it not only accommodates in-the-wild images but also, to a certain extent, grasps previously unseen visual instructions.\n\nConsequently, once a user prepares an example alongside its corresponding retouching instance, they can efficiently and effortlessly apply this operation to a similar collection of images. By grasping the inherent editing intention within the provided instruction, the proposed model becomes a valuable tool in executing analogous creative tasks on pertinent target images.\n\n\n**_Evaluation metric contribution._**\n\nWe appreciate the suggestion to tune down the introduced evaluation metric in our manuscript. We acknowledge that proposing a new metric entails thorough user studies and substantial evidence to establish its alignment with human perception. In our study, the proposed evaluation metric solely serves as an initial endeavor to enhance the assessment of this novel generative task. We will proceed to revise our manuscript to more accurately convey the significance of the evaluation metric.\n"}, {"review_id": "MI5iLWyI2n", "reviewer": "Reviewer_uVk2", "summary": "The authors propose a novel, visual-based method for image editing, using a pair of example images as instructions for users to specify the desired operations. The proposed idea uses the details and nuance in the example pairs to accomplish complex editing goals. The authors formulate this approach as a diffusion inpainting task, where the instruction pair and the query image are given and tiled as a grid. To achieve the desired quality, the visual prompt and region of user interest are injected into the diffusion structure as conditions, and the whole pipeline is trained as a latent diffusion model with classifier-free guidance. The authors also developed a metric to assess the quality of manipulations performed and conducted quantitative and qualitative studies.", "questions": "1. It is unclear how the image grid, Xt, is represented. Figure 2 shows that only I' is converted to noise after the forward diffusion process. However, in LDMs, the diffusion process is in the z space, where the whole image is encoded.\n\n2. Is resolution one of the limitations of this method since the inpainting area is only 1/4 of the input image grid?\n\n3. How the model is trained using the four datasets mentioned in 4.1 is unclear. Is it trained from scratch? \n\n4. I am curious about how necessary the inpainting setup is. For example, what will the model perform if the task is to generate only I' instead of the whole grid (in other words, the visual clues and instructions are comping from cross-attention only)?  \n\n5. Have the authors tried incorporating the region of interest using different methods (such as masking in latent space similar to eDiff-I)?", "limitations": "Yes, the limitations are addressed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method demonstrates a new way to manipulate images using visual instructions only.\n2. The model generalizes well on downstream tasks compared to other methods requiring task-specif models.\n3. The authors propose a new metric to evaluate image manipulation tasks.", "weaknesses": "1. For many common image editing tasks, it will be challenging or tedious for users to find the example pairs. For example, if a user wants to convert a photo into a painting by a specific artist, preparing the paired picture for the given painting will be hard without using existing pre-trained style-transfer models.\n\n2. The authors think one of the disadvantages of using text prompts is that text prompts are abstract. However, the proposed method is also abstract if users want specific, exact manipulations. For example, if we're going to make the spout of a teapot 2 cm longer or want a particular proportion of the teapot shape, finding the pair examples and ensuring the model does the exact manipulations will be challenging.\n\n3. Conducting user studies to support the assumptions would be great. For example, is it challenging for the users to find the pairs for their desired editing? Do the results look better to them (FID and CLIP scores are shown not to reflect human preferences/aesthetics in prior works like [1])?\n\n[1] Better Aligning Text-to-Image Models with Human Preference by Wu et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\n**_Difficulty in finding example pairs._**\n\nAs stated in the common response, we highlight that the practical application of our proposed system is to offer an efficient tool for users to automatically streamline their desired editing operation to a new-coming image, which acts like the Format Painter in Office Software, rather than targeting editing a specific image. Specifically, it is natural and intuitive to instruct an image with language considering its easy access. But when it comes to a situation where users want to apply analogous manipulation referring to his or others’ finished edits. In this case, it is particularly challenging to tackle with text condition or single image conditioned diffusion models because it requires inducing the underlying operations and imposing the intended manipulation on the target image. Furthermore, since the proposed system behaves like a ImageBrush, users are capable of accomplishing this solely with a pair of images without recording tedious settings such as a series of parameters in Photoshop.\n\nTo sum it up, users could always manually create or find their preferred visual instructions, with which our model enables them to conveniently leverage the implicit operations. Admittedly speaking, we agree that this framework will be more user-friendly if it could support visual instructions without relying on spatial alignment of the example image. Regarding how to deal with more versatile visual prompts, we will leave it for future research.\n\n\n**_Challenges in Precise Manipulations._**\n\nWe value the insightful observation made by the reviewer. Indeed, finely manipulating the shape of a teapot presents challenges for both language-based and visual instructions. However, in situations involving unarticulated object appearances and artistic styles, our proposed system exhibits advantages over text-based guidance. Moreover, our model's capabilities enable novel applications as elaborated in the last section.\n\nFor scenarios where users require exact or specific manipulations, such as extending the spout of a teapot by 2 cm or adjusting the proportions of the teapot shape, it might be beneficial to combine both visual instructions and text prompts to guide the image manipulation process. In this case, the visual instruction can provide the overall style or appearance information, while the text prompt can convey the specific details or numeric parameters. Integrating both modalities can potentially improve the accuracy and precision of the manipulation while maintaining the advantages of visual instructions in capturing complex concepts.\n\nFurthermore, we acknowledge that finding suitable pair examples for highly specific manipulations can be challenging. One potential solution to this issue is to develop an interactive system that allows users to provide iterative feedback or refine the visual instructions during the manipulation process, which could help the model better understand and satisfy the user's requirements.\n\n**_Conducting User Studies._**\n\nThank you for your suggestion. We will consider incorporating this type of user study as part of our future work, as well as addressing the limitations of metrics such as CLIP scores and FID in our evaluation.\n\n\n**_Unclear about diffusion process._**\n\nIn LDMs, the entire z space is subject to the addition of noise. In the reverse process at each timestep, we re-inject clean $\\textbf{E}$, $\\textbf{E}'$, and $\\textbf{I}$ into the UNet to provide a clean context, allowing the noise prediction task to focus on recovering the inpainting task of $\\textbf{I}'$. \n\n**_Unclear about image grid and resolution problems._**\n\nCurrently, this limitation exists wherein we can generate results only at a resolution of 256x256, despite the model's inherent capacity to produce outputs at 512x512.  We remain optimistic about overcoming this constraint in the future. One potential solution could involve concatenating the inputs channel-wise, a strategy that would preserve resolution without compromise.\n\n\n**_Is it trained from scratch?_**\n\n We initialize our model with the latent diffusion weights.\n\n\n**_How necessary the inpainting setup is._**\n\nOur primary motivation for constructing the inpainting task is to inject low-level features from $\\textbf{E}$, $\\textbf{E}'$, and $\\textbf{I}$ through the self-attention branch rather than relying solely on cross-attention. If we do not construct this inpainting task and solely incorporate features into the UNet's cross-attention to control diffusion, the image features would lose a significant amount of detail after being encoded by CLIP. Moreover, since the injection takes place in the latent space of LDMs, low-level information must be restored using the knowledge of the VAE's encoder.\n\nBy using the grid-like image inpainting approach, features of $\\textbf{E}$, $\\textbf{E}'$, and $\\textbf{I}$ are fully aligned with the predicted image $\\textbf{I}'$, as they all pass through the VAE's encoder. This method allows us to retain as much low-level information as possible in a simple and effective manner. If the loss of resolution is a concern, we can adopt a channel concatenation approach (however, this would require allocating separate parameters for each image to avoid alignment issues).\n\n**_Incorporating the region of interest using different methods._**\n\nWhile we have not explicitly experimented with this method in our current work, it could be an interesting direction for future research. Masking in the latent space or re-weight the attention matrix may potentially lead to further improvements in image generation by more accurately preserving the user's region of interest and better aligning with the provided instructions. We appreciate the reviewer's suggestion and will try incorporating those approaches in future studies to assess their efficacy.\n\n"}, {"review_id": "sBHOvHXNhe", "reviewer": "Reviewer_fTo3", "summary": "This paper proposed a novel scheme for image editing via exemplar-based.\nAn image edit will be represented as an in-context instructions.\nTo that end, the novel framework is trained to perform image editing as image in-painting task.\nExperiment results indicate the advantages of this scheme over typical text-guided image editing frameworks.", "questions": "My initial rating is \"Borderline accept\" as I believe that while this paper is well-motivated, there are several revisions that need to be made.\n\nIn addition to the questions raised in the Weakness section, I also have some technical inquiries:\n\n- Is the output size of all images 256x256? (which is smaller than other methods typically using 512x512).\n- Was the model fine-tuned from Latent Diffusion, or trained from scratch?", "limitations": "The author has briefly addressed limitation. As a minor note, I recommend authors to include some qualitative results for limitation or challenging cases to encourage future research.\n\nI strongly urge the authors to consider two key improvements during the revision process:\n\n(1) If the authors intend to concentrate on image editing, it would be advisable to narrow down their focus to a single specific task. This would enable a more comprehensive exploration of the chosen task and provide greater clarity to the readers.\n\n(2) If the authors aim to emphasize the modality aspect, I recommend conducting a more extensive comparison with existing works. This would help establish the unique contributions of the proposed method and demonstrate its superiority over previous approaches.\n\nRegardless of the chosen direction, I believe that implementing these revisions would significantly enhance the paper's quality.\n\n---\n\n**After Rebuttal**:\n\nI have read authors' rebuttal and other reviews. I thank authors for efforts to further address my concerns.\n\nAfter careful consideration, I decided to maintain my rating as 5-Borderline accept.\n\nMy justification for not higher rating are:\n\n1. The contribution about User's Interface (hybrid context injection strategy) is not (yet) fully validated.\n1. Overlapping claims with Visual Prompting [1] paper (Also raised by [Reviewer zhtQ](https://openreview.net/forum?id=EmOIP3t9nk&noteId=d3daWKNPI3))\n1. Some Figure, Table need clearer explanation or captions (Also raised by [Reviewer RQEf](https://openreview.net/forum?id=EmOIP3t9nk&noteId=SRf3GZGiV2))\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality/Novelty**:\n- At a high level, this paper proposes a novel protocol for image editing: visual prompting. I believe this is an area will attract interests of both academic and applications.\n- The designed framework, called ImageBrush, enable performing image manipulation in-context learning approach. Moreover, the interface design caters to user interest, which, although an add-on feature, proves beneficial for end-users.\n- Through experiments and visual example, the paper demonstrates the merits of proposed protocol, which can be more acurate and efficient for image editing tasks.\n- The authors did give efforts to try on multiple types of manipulation (e.g., seketon to fashion pose), instead of only standard image editing task.\n\n**Presentation/ Writing**:\n- Overall, the paper is well-organized and easy to follow.\n- Framework illustration is fairly good; qualitative examples depicts the merits of the method.", "weaknesses": "**1. Some parts need to be revised to avoid over-claiming/ clearer**\n\n1.1 The contribution mentioned in lines 72-73 overlaps with the Visual Prompting approach proposed in [1].\nWhile I understand that this paper focuses on more specific tasks such as image editing and pose transfer, it would be advisable to use a more restrained tone.\nInstead of making strong claims about the original idea of visual prompting, the authors could emphasize the concepts of \"progressive inpainting\" or \"prompting for additional tasks.\"\n\n1.2 The \"Related Work\" section should include references to \"Visual Prompting\" or \"In-context learning\" to provide more complete survey of prior work.\n\n1.3. Figure 3 is unclear and requires further clarification. Please explain this Figure in the rebuttal.\n\n**2. Flaws in experiments**\n\n2.1. While the main contribution of this paper is an in-context learning scheme for image manipulation, I do not see any comparisons against existing in-context learning scheme (e.g., Visual Prompting [1] or Painter [2]).\n\nI am not against the fact that the authors demonstrate the effective against \"state-of-the-art\" methods for each specific tasks (e.g.: compare with InstructPix2Pix in image editing).\nI believe that it's crucial to compare against in-context learning methods as well.\nFor example: Authors should compare image in-painting task with Visual Prompting [1].\n\n2.2. Lack of results for Interface design.\nWhile I second this contribution, there is no qualitative results for this module in the main paper.\nI have checked the Supplementary, only one example (edit of \"putting a crown on the woman\" is included). This example is not convincing itself, as I see the the crown in both approach is still not visually similar to the given crown in visual prompting.\n\nAdditionally, I feel like these results can be generated by different noises. For example, given the same visual prompt (e.g., \"Put a crown on the woman\", different noises can also lead to different visually different crown). For that reason, while I second this module, I am not convinced that this module works in practice.\n\n2.3. Unfair comparison in video inpainting task.\nFollowing [4] Video-FID (together with PSNR and SSIM) is used to evaluate the methods. Is it better to use Video-FID instead of FID?\n\n**3. Miscellaneous**\n\n3.1 Line 124- 138: While cite Latent Diffusion model [5], the formula are in pixel-space. Should the citation update to DDPM instead to suit the following formular?\n\n3.2 Line 243: Missing citation for the \"CLIP Direction Similarity\" mentioned.\n\n3.3 Line 218: Is the dataset name correct? My understanding is the authors indeed use the [InstructPix2Pix's CLIP-filtered-dataset]((https://github.com/timothybrooks/instruct-pix2pix#generated-dataset). There might be a misunderstood, as in InstructPix2Pix, \"LAION Improved Aesthetics 6.5+\" is used as just a small part of generating the final dataset (as written in [this](https://github.com/timothybrooks/instruct-pix2pix#11-manually-write-a-dataset-of-instructions-and-captions)).\n\n**References**\n\n[1] Amir et al., *Visual Prompting via Image Inpainting*, NeurIPS 2022.\n\n[2] Xinlong et al., *A Generalist Painter for In-Context Visual Learning*, CVPR 2023.\n\n[3] InstructPix2Pix et al., *InstructPix2Pix: Learning to Follow Image Editing Instructions*, CVPR 2023.\n\n[4] Xueyan et al., *Progressive Temporal Feature Alignment Network for Video Inpainting*, CVPR 2021.\n\n[5] Robin et al., *High-Resolution Image Synthesis with Latent Diffusion Models*, CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\n**_1. Some parts need to be revised to avoid over-claiming/ clearer_**\n\n1.1-1.2. We greatly appreciate the constructive advice provided on our writing. To ensure accuracy and clarity, we will carefully tune down the claim and provide a more comprehensive introduction to the \"In-Context Learning\" work in our future version.\n\n1.3. We compare our method with InstructPix2pix from two perspectives: aligning generated results with instructions (higher CLIP direction similarity) and retaining query image identity (higher image similarity). However, these two metrics can conflict to some extent, as not editing the query image would lead to perfect image similarity. Therefore, we separately measure them while controlling another variable, which involves averaging CLIP direction similarity for samples within the same CLIP image similarity bin and vice versa.\n\n\n**_2. Flaws in experiments_**\n\n2.1 We further include additional comparison results against VisualPrompt in our appended PDF document. Fig.4 illustrates the qualitative results, revealing that ImageBrush yields plausible and realistic edits whereas VisualPrompt tends to produce conservative predictions and struggles on instruction comprehension. We believe this advantage can be attributed to the progressive synthesis nature of Diffusion. Moreover, quantitative results in Table.1 demonstrate the superiority of our approach against VisualPrompt on most tasks.\n\n2.2 We have additionally incorporated more qualitative examples of our interface design in the attached PDF file. Fig. 1 depicts diverse examples with distinct emphasis on visual instructions. For example, synthesis of additional beards when the beard area is emphasized, the generation of more detailed hats when a bounding box is applied to the hat part, and the improved ability of our model to understand and execute the intention of editing a bucket to black when the emphasis is on the bucket. Furthermore, when a small object is added, our model successfully comprehends and synthesizes an object with the assistance of this input. Similarly, by placing emphasis on the background, more trees are synthesized in that region.\n\n2.3 We provide further comparisons utilizing standard metrics, including PSNR, SSIM, and LPIPS, for the video inpainting task, as displayed in Table 1. Across these pixel-aligned metrics, TSAM demonstrates superior performance. This result can be attributed to the tendency of diffusion-based methods to over-imagine details. However, a higher FID value in our method indicates superior photorealism and a more pleasing visual quality.\n\n**_3. Miscellaneous_**\n\nThank you for bringing this to our attention. We will ensure the citation is updated to DDPM for accuracy. Additionally, we will incorporate the proper citation for \"CLIP Direction Similarity\". As for the dataset name, we confirm that we utilize InstructPix2Pix's CLIP-filtered dataset, and we will proceed to modify it to ensure consistency.\n\n**_4. Questions_**\n\n4.1 Regarding the image resolution, all images are of size 256x256, which is smaller than the typical size of 512x512. This is because our model relies on concatenating images to learn low-level appearance correlation via the self-attention mechanism, resulting in additional spatial dimension consumption.\n\n4.2 We initialize our model with the latent diffusion weights.\n"}]

Author rebuttal:
We express our gratitude to the reviewers for their valuable insights and suggestions. In this section, we address a common concern raised by the reviewers regarding the task formulation.

In this study, we introduce a novel protocol for image manipulation based on visual instructions. As indicated by our title, our objective is to offer users a rapid and efficient tool for implementing desired edits. This tool operates akin to an ImageBrush, streamlining the process of customizing batches of images. This paradigm holds promising potential for various innovative applications. For instance, photographers can seamlessly apply Photoshop-style retouching to entire collections of similar images, enhancing their workflow. Additionally, users who encounter compelling editing instances from peers or pre-trained models can effortlessly apply these modifications to their own images, eliminating the need for original intricate intermediate steps or specific parameter adjustments.

According to the reviewers feedback, the major concern is that finding the intended visual instruction examples might be challenging. ​​However, it's important to note that this challenge is not insurmountable. On the one hand, many paired instruction examples could be obtained through direct manipulation using software like Photoshop. On the other hand, our model exhibits a degree of flexibility in accommodating variations between query images and the provided instruction examples, as evidenced in Figures 2 and 3 within the attached PDF document.


Meta-review: The paper received overwhelmingly positive reviews, with the majority of reviewers recommending acceptance. 
The main negative points raised in the reviews were in connection with missing comparison with specific previous methods and additional metrics, which were then included after the rebuttals in a way that most reviewers indicated as satisfactory.
Other outstanding doubts have to do with some constraints of visual prompting and editing workflows that use this technique more than weaknesses of this work specifically, and seems to also have been satisfactorily addressed by the rebuttals.
Reviewers praised the innovation of this work, which includes the introduction of a novel evaluation metric for exemplar-based image manipulation that does require human evaluation, the quality of the results of the empirical downstream evaluations and comparisons with alternative methods where feasible, and the exemplar-based visual prompting that it opens up.
Decision: Accept (poster)