SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison Methods. To thoroughly evaluate the effectiveness of our approach, we compare it against other state-of-art models tailored for specific tasks, including: VisualPrompt [4], the pioneer study in visual in-context learning; SDEdit [33], a widely-used stochastic differential equation (SDE)-based image editing method; Instruct-Pix2pix [6], a cutting-edge language-instructed image manipulation method; CoCosNet [72], an approach that employs a carefully designed correspondence learning architecture to achieve exemplar-based image translation; and TSAM [73], a model that incorporates a feature alignment module for video inpainting.
Evaluation Metrics. In the image manipulation task, it is crucial for the edited image to align with the intended direction while preserving the instruction-invariant elements in their original form. To assess the degree of agreement between the edited image and the provided instructions, we utilize a cosine similarity metric referred to as CLIP Direction Similarity in the CLIP space. In order to measure the level of consistency with original image, we utilize the cosine similarity of the CLIP image embedding, CLIP Image Similarity, following the protocol presented in [6]. Additionally, for other generation tasks such as exemplar-based image translation, pose transfer, and video in-painting, we employ the Fréchet Inception Score (FID) [18] as an evaluation metric. The FID score allows
Figure 3: Quantitative Comparison on Image Editing. We compare our approach with the representative text-instruction method in terms of direction consistency and similarity consistency.
us to evaluate the dissimilarity between the distributions of synthesized images and real images. Following similar protocol with [73], the V-FID is also included to take into account the temporal consistency in video inpainting. For image generation, we introduce the SSIM [63] metric to further assess the structural similarity and visual quality.
Image Generation Paradigm. In the image manipulation task, a pair of transformed images is utilized as visual instructions to edit a target image within the same instruction context. For other tasks, we conduct experiments in a cross-frame prediction setting, where we select two temporally close frames and use one of them as an example to predict the other. Specifically, we employ one frame along with its corresponding label (e.g., edge, semantic segmentation, keypoints, or masked image) as a contextual example, and take the label from another frame as a query to recover that frame. To obtain the label information, we extract keypoints from the UBC-Fashion dataset using OpenPose [8], and for the LRW dataset, we utilize a face parsing network [68] to generate segmentation labels. To ensure that the selected frames belong to the same context, we restrict the optical flow between them to a specific range, maintaining consistency for both training and testing. It is important to note that, for fair comparison, we always utilize the first frame of each video as a reference during the evaluation of the video inpainting task.
Evaluation Results. The comparison results for image manipulation are presented in Fig. 3. We observe that our results exhibit higher image consistency for the same directional similarity values and higher directional similarity values for the same image similarity value. One possible reason is the ability of visual instructions to express certain concepts without a modality gap, which allows our method to better align the edited images.
To demonstrate the versatility of our model, we conducted experiments using in-the-wild videos that encompassed diverse real-world contexts. The results for various downstream tasks are presented in Table 1, showing the superior performance of our approach across most datasets. It is noteworthy that our model achieves these results using a single model, distinguishing it from other methods that require task-specific networks. In inpainting task, our model attains a higher FID score while yielding a comparatively lower Video-FID score and SSIM value. We speculate that it is because our framework lack explicit temporal awareness such as optical flow in [73].
We provide a qualitative comparison with SDEdit [33] and Instruct-Pix2pix [6] in Fig. 4. In the case of the SDEdit model, we attempted to input both the edit instruction and the output caption, referred to as SDEdit-E and SDEdit-OC, respectively. In contrast to language-guided approaches, our method demonstrates superior fidelity to the provided examples, particularly in capturing text-ambiguous concepts such as holistic style, editing extent, and intricate local object details. Comparison with VisualPrompt [4] is depicted in Fig. 6 where we achieve more realistic and plausible editing results.
Additionally, we provide a qualitative analysis on a real-world dataset in Fig. 5. Compared to CocosNet [72], our approach demonstrates superior visual quality in preserving object structures, as seen in the cabinet and table. It also exhibits higher consistency with the reference appearance, particularly noticeable in the woman's hair and dress. Furthermore, our method achieves improved shape consistency with the query image, as observed in the mouth. When compared to TSAM [73], our
The picture is of a street in Paris
The tiger cub is a snow leopard
Turn it into a snowy landscape
Turn the bridge into a bridge made of ice
Figure 4: Qualitative Comparison on Image Manipulation. In contrast to language-guided approaches, our editing results guided by visual instruction exhibit better compliance with the provided examples.
Figure 5: Qualitative Comparison on In-the-wild Dataset. We conduct experiments on various downstream tasks including exemplar-based image translation, pose transfer and inpainting.
approach yields superior results with enhanced clarity in object shapes and more realistic background filling. These improvements are achieved by effectively leveraging visual cues from the examples.
Novel Evaluation Metric. Exemplar-based image manipulation is a novel task, and the evaluation metric in the literature is not suitable for this task. Therefore, we present an evaluation metric
Figure 6: Comparison with VisualPrompt [4] on Image Editing. Our model achieves plausible and realistic edits whereas VisualPrompt tend to produce conservative prediction and struggle on instruction comprehension.
Figure 7: Case Study in Terms of Prompt Fidelity and Image Fidelity. The prompt fidelity and image fidelity are calculated between paired images with dashed yellow banner and dashed blue banner, respectively.
that requires neither ground-truth image nor human evaluation. Specifically, our model, denoted as  $\Phi_{\theta}$ , takes the examples  $\{\mathbf{E}, \mathbf{E}'\}$ , and the source image  $\mathbf{I}$  to produce a manipulated output  $\mathbf{I}'$ . This procedure is presented as  $\mathbf{I}' = \Phi_{\theta}(\mathbf{I}|\mathbf{E}' \to \mathbf{E})$ . The goal is to let the output image  $\mathbf{I}'$  abide by instruction induced from examples  $\mathbf{E}$  and  $\mathbf{E}'$  while preserving the instruction-invariant content of input image  $\mathbf{I}$ . We define prompt fidelity to assess the model's ability to follow the instruction. According to the symmetry of  $\{\mathbf{E}, \mathbf{E}'\}$  and  $\{\mathbf{I}, \mathbf{I}'\}$ , if its operation strictly follows the instruction, the model should manipulate  $\mathbf{E}$  to  $\mathbf{E}'$  by using  $\mathbf{I} \rightarrow \mathbf{I}'$  as the prompt (see pictures labeled with a dashed yellow tag in Fig. 7). We express the prompt fidelity as follows
$$
\Delta_ {\text {p r o m p t}} = \operatorname {F I D} \left(\mathbf {E} ^ {\prime}, \Phi_ {\theta} \left(\mathbf {E} \mid \mathbf {I} \rightarrow \mathbf {I} ^ {\prime}\right)\right). \tag {7}
$$
On the other hand, to evaluate the extent to which the model preserves its content, we introduce an image fidelity metric. If  $\mathbf{I}'$  maintains the content of  $\mathbf{I}$ , the manipulation should be invertible. That is to say, by using  $\mathbf{E}' \rightarrow \mathbf{E}$  as the prompt, the model should reconstruct  $\mathbf{I}$  from  $\mathbf{I}'$  (see pictures labeled with a dashed blue tag in Fig. 7). Therefore, we define the image fidelity as
$$
\Delta_ {\text {i m a g e}} = \operatorname {F I D} (\mathbf {I}, \Phi_ {\theta} \left(\mathbf {I} ^ {\prime} \mid \mathbf {E} ^ {\prime} \rightarrow \mathbf {E}\right)). \tag {8}
$$
Ablation Study. We performed ablation studies on three crucial components of our method, namely the diffusion process for context exploitation, the vision prompt design, and the injection of human-interest area. Specifically, we conducted experiments on our model by (1) replacing the diffusion process with masked image modeling, (2) removing the cross-attention feature injection obtained from vision prompt module, and (3) deactivating the input of the human interest region. Specifically, we implemented the masked image modeling using the MAE-VQGAN architecture following the best setting of [4]. The numerical results on image manipulation task are shown in Table 2. The results demonstrate the importance of the diffusion process. Without it, the model is unable to progressively comprehend the visual instruction and refine its synthesis, resulting in inferior generation results. When we exclude the feature integration from the visual prompt module, the model struggles to understand high-level semantics, leading to trivial generation results on these instructions. Furthermore, removing the region of interest interface results in a slight decrease in performance, highlighting its effectiveness in capturing human intent.
Case Study. In Fig. 7, we present two cases that showcase our model's performance in terms of image and prompt fidelity. The first case, depicted on the left, highlights our model's capability to
Table 2: Ablation Study on Image Manipulation. Here we evaluate on our proposed metric.
w/o Diffusion	w/o Cross-Attention	w/o Interest Region	Full Model
Prompt Fidelity	78.65	39.44	24.29	23.97
Image Fidelity	82.33	41.51	25.74	24.43
Figure 8: Generation Results by Customized Prompts. The regions of interest are indicated by purple box and generated results are displayed within red box. From left to right we illustrate cases where emphasis is placed on beards, hats, buckets, flying objects and trees.
Figure 9: Image Editing with Varied Queries. We showcase edited results corresponding to three distinct queries. It covers a variety of query contexts, demonstrating changes in layout, object quantity, weather, pose, shape and perspective.
utilize its predicted result  $\mathbf{I}'$  to reconstruct  $\mathbf{I}$  by transforming the image back to a dark appearance. With predicted  $\mathbf{I}'$ , our model also successfully edits the corresponding example from  $\mathbf{E}$  to  $\mathbf{E}'$ , making the example image colorful. On the right side of the figure, we present a case where our model encounters difficulty in capturing the change in the background, specifically the disappearance of the holographic element.
Effect of User Interface. Fig. 8 depicts diverse examples with distinct emphasis on visual instructions. For example, synthesis of additional beards when the beard area is emphasized, the generation of more detailed hats when a bounding box is applied to the hat part, and the improved ability of our model to understand and execute the intention of editing a bucket to black when the emphasis is on the bucket. Furthermore, when a small object is added, our model successfully comprehends and synthesizes an object with the assistance of this input. Similarly, by placing emphasis on the background, more trees are synthesized in that region.
Image Editing with Diverse Queries. We present a collection of qualitative examples encompassing diverse query contexts in Fig. 9. The results suggest our model's competence in generating coherent and reliable outcomes across various query scenarios.
In this article, we present an innovative way of interacting with images, Image Manipulation by Visual Instruction. Within this paradigm, we propose a framework, ImageBrush, which holds promising potential for various applications.
Limitation and Future Work. 1) The proposed model may face challenges when there is a substantial disparity between the given instruction and the query image. 2) The current model also encounters difficulties in handling intricate details, such as subtle background changes or adding small objects. 3) Future work can delve into a broader range of datasets and tasks, paving the way for future generative visual foundation models.
[71] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[72] Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. Cross-domain correspondence learning for exemplar-based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5143-5153, 2020.
[73] Xueyan Zou, Linjie Yang, Ding Liu, and Yong Jae Lee. Progressive temporal feature alignment network for video inpainting. In CVPR, 2021.
A Analysis of In-Context Instruction with Multiple Examples
Our approach can be naturally extended to include multiple examples. Specifically, given a series of examples  $\{\mathbf{E}_1,\mathbf{E}_1',\dots ,\mathbf{E}_n,\mathbf{E}_n',\mathbf{I}\}$ , where  $n$  represents the number of support examples, our objective is to generate  $\mathbf{I}'$ . In our main paper, we primarily focused on the special case where  $n = 1$ . When dealing with multiple examples, we could also establish their spatial correspondence by directly concatenating them as input to our UNet architecture. Specifically, we create a grid  $x_{t - 1} = \mathrm{Grid}(\{\mathbf{E}_1,\mathbf{E}_1',\dots ,\mathbf{E}_n,\mathbf{E}_n',\mathbf{I}\}_{t - 1})$  that can accommodate up to eight examples following [4]. To facilitate contextual learning, we extend the input length of our prompt encoder  $e_p$ , and incorporate tokenized representations of these collections of examples as input to it. In this way, our framework is able to handle cases that involve multiple examples.
Below we discuss the impact of these examples on our model's final performance by varying their numbers and orders.
Figure 10: Analysis of In-Context Instruction with Multiple Examples.
A.1 Number of In-Context Examples.
In our dataset, which typically consists of 4 examples, we examine the influence of the number of in-context examples by varying it from 1 to 3, as illustrated in Figure 10. We evaluate this variation using our proposed metrics: prompt fidelity  $\Delta_{\mathrm{prompt}}$  and image fidelity  $\Delta_{\mathrm{image}}$ , which assess instruction adherence and content consistency, respectively. The results demonstrate that increasing the number of support examples enhances performance, potentially by reducing task ambiguity.
Furthermore, we present additional instruction results in Figure 11, showcasing the impact of different numbers of in-context examples. It is clear that as the number of examples increases, the model becomes more confident in comprehending the task description. This is particularly evident in the second row, where the synthesized floor becomes smoother and the rainbow appears more distinct. Similarly, in the third row, the wormhole becomes complete. However, for simpler tasks that can be adequately induced with just one example (as demonstrated in the last row), increasing the number of examples does not lead to further performance improvements.
A.2 Order of In-Context Examples.
We have also conducted an investigation into the impact of the order of support examples by shuffling them. Through empirical analysis, we found no evidence suggesting that the order of the examples has any impact on performance. This finding aligns with our initial expectation, as there is no sequential logic present in our examples.
B More Visual Instruction Results
In this section, we present additional visual instruction results, where the synthesized result is highlighted within the red box. These results demonstrate the effectiveness of our approach in handling diverse manipulation types, including style transfer, object swapping, and composite operations, as showcased in Fig. 12 and Fig. 13. Furthermore, our method demonstrates its versatility across real-world datasets, successfully tackling various downstream tasks, such as image translation, pose translation, and inpainting in Fig. 14 and Fig. 15. Importantly, it is worth noting that all the presented results are generated by a single model.
Figure 11: Qualitative Analysis on Number of In-Context Examples.
Figure 12: Image Manipulation Results by Visual Instruction.
Figure 13: Image Manipulation Results by Visual Instruction.
Figure 14: Instruction Results on In-the-wild Dataset.
Figure 15: Instruction Results on In-the-wild Dataset.