IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks
Authors: Jiarui Xu
Yossi Gandelsman
Amir Bar
Jianwei Yang
Jianfeng Gao
Trevor Darrell
Xiaolong Wang
Venue: Accepted by TMLR ()

Abstract: In-context learning allows adapting a model to new tasks given a task description at test time. In this paper, we present IMProv - a generative model that is able to in-context learn visual tasks from multimodal prompts. Given a textual description of a visual task (e.g. “Left: input image, Right: foreground segmentation”), a few input-output visual examples, or both, the model in-context learns to solve it for a new test input.  We train a masked generative transformer on a new dataset of figures from computer vision papers and their associated captions, together with a captioned large-scale image-text dataset. During inference time, we prompt the model with text and/or image task example(s) and have the model inpaint the corresponding output. We show that training our model with text conditioning and scaling the dataset size improves in-context learning for computer vision tasks by over $+10\%$ AP for Foreground Segmentation, over $+5\%$ gains in AP for Single Object Detection, and almost $20\%$ lower LPIPS in Colorization. Our emperical results suggest that vision and language prompts are complementary and it is advantageous to use both to achieve better in-context learning performance.

Published in Transactions on Machine Learning Research (09/2024)
IMProv: Inpainting-based I Prompting for Com- puter Vision Tasks
Jiarui Xu UC San Diego
jiz026 edu
Yossi Gandelsman UC Berkeley
yossid@gandelsman.com
Amir Bar UC Berkeley
amér. immrrbbar@berkeleey. edu
Jianwei Yang Microsoft Research
jianwei. somnweii.yng@mmicrsssff.toom
Jianfeng Gao Microsoft Research
ifgoo@microsoftt.com com
Trevor Darrell UC Berkeley
treeordarrell@Beekkkee edu
Xiaolong Wang
riw0 I 12 edu
UC San Diego
Reviewed OpenReview: https: opwnreeveee net/ formm? id- gBTgkk2HAf
Abstract
In-context learning allows adapting model to new tasks given a task description at test time. In this paper, we present IMProv generative model that is able to in-context learn visual tasks from multimodal prompts. Given a textual description of a visual task (e.g. "Left: input image, Right: foreground segmentation") "), a few input-output visual examples, or both, the model in-context learns to solve for new test input. We train masked generative transformer new dataset of figures from computer vision papers and their associated captions, together with captioned large-scale image-text dataset. During inference time, prompt the model with text and/or image task example(s) and have the model inpaint the corresponding output. We show that training our model with text conditioning and scaling the dataset size improves in-context learning for computer vision tasks by over +10% AP for Foreground Segmentation, over +5% gains AP for Single Object Detection, and almost 20% lower LPIPS in Colorization. Our emperical results suggest that vision and language prompts complementary and it is advantageous learning performance.
both achieve better in-context
1 Introduction
In-context learning (ICL) (Brown et al., 2020; Chan et al., 2022; Xie et al., 2021), also known as few-shot prompting, is an exciting new paradigm in machine learning that allows a model to adapt to novel downstream tasks without fine-tuning or changing the model's weights. In natural language processing (NLP), ICL is considered an emergent property of large language models (Brown al., 2020; Touvron al., 2023; Chowdhery et al., 2022) and it was first introduced in the seminal paper of GPT-3 (Brown et al., 2020). A few-shot, prompt typically includes examples of (input, output) pair(s). The few-shot performance of large

---

Published in Transactions on Machine Learning Research (09/2024)
(a) Visual Prompt
(b) Textual Prompt
[c) Multimodal Prompt
"Laft
input image,
Right
Right Black and white
Bregrroun/backgrrooo
foreground/backer
segnentation an airplane"
segnentation airplane"
"Right output image Yellow Depth rap""
"Left 1ropus 1nage, Right Edge maр The Best..."
"Rignt Photograph
image
"Right outtuu 1iogee
1ngut inage, Right Scmantic seg-
1nogut inage, Right Dopth map Spiiay Pix...
Cabin
Matuga Country
Barn?
mwtnutlon"
Figure 1: Inpainting- based Multimodal Prompting for vision (IMProv). Top: Our model in-context learns to solve computer vision tasks by inpainting the masked area with the task solution (shown in red square) using visual onput- examples (a), a textual task description (b), or both (c). Bottom: IMProv prediction examples.
language models has been shown to achieve competitive results on NLP tasks, sometimes surpassing prior fine-tuning approaches (Brown et al., 2020).
In computer vision, the full potential of in-context learning (ICL) is still far from being realized. To enable model to perform in-context learning during test time, there are two key challenges that need to be addressed. Firstly, the model's architecture should be designed in such a way that it can effectively process prompts from various vision tasks. This means it should be capable of receiving task instructions and/or input-output examples as inputs to the model. Secondly, a different approach to training these models is required. While in natural language processing (NLP), the emergence of ICL has been facilitated by utilizing large-scale, non-curated data, in computer vision, even generative models trained on billions of non-curated text-image pairs have failed to achieve similar results.
A possible approach to enable test-time few-shot prompting for computer vision tasks is to train multi-task inpainting model (Wang et al., 2022; 2023b; Bar et al., 2022). For example, previous approaches (Wang et al., 2022; 2023b) adopted a fully supervised paradigm to train a model over a prodetermined set of vision tasks. However, this line study requires a handful of manual annotations and thus struggles to scale and generalize well to unseen vision tasks. Instead of explicitly designing the tasks, Bar al. (2022) took a different. unsupervised approach by proposing learn from unstructured Computer Vision Figures data, where images have implicit task supervision and grid-like structure. However, using vision-only prompting (Bar et al., 2022) sulfers from ambiguities and is limited in its ability to describe specific visual task.
To alleviate these difficulties, we propose to multimodal ICL by prompting using input that consists of both pixels and text. Intuitively, these two modalities can work in synergy to enhance the understanding of the world and its complexities. For example, during a conversation, people use language to communicate ideas and vision perceive facial expressions and conversational gestures (Cassell et al., 1999; McNeill, 2019). For prompting, conditioning vision models on text can enable describing instructions in an efficient manner and reduce ambiguities without the necessity for multiple high-quality visual examples.
Equipped with this intuition, we train model, dubbed IMProv, to inpaint randomly masked regions given the rest of the image and a caption as context Our training does not require erplicil definitiioss of tasks and annotations for each task. To demonstrate multimodal learning can be boosted by larger dataset, we collected new dataset of imago-text pairs from Semantic Scholar, which three times larger than the largest existing computer vision figures dataset. We train new model by performing inpainting on randomly masked images from combination ion of the newly constructed data and LAION 400M (Schuhmann et. al., 2021).
2

---

Published in Transactions on Machine Learning Research (09/2024)
Cross Entropy Loss
Pretrained and frozen
Trained from scratch
Reconstructed tokens
VQGAN Encoder
ViT Decoder
CLIP Text Encoder
ViT Encoder
Picture: These people are not real they pro- duced by our gener- ator that allows..
Input image
Random (non-masked) input patches
Input Text
Figure 2: IMProv Architecture. During training, the input image is patchified, masked, and fed to the model together with the associated caption CLIP (Radford et al., 2021) embeddings. For each masked token, the decoder outputs a distribution over a frozen pretrained VQGAN (Esser et al., 2021) codebook. The model trained with cross-entropy loss.
At test-time, our model exhibits emerging capabilities such as zero-shot prompting for vision tasks, e.g., performing foreground segmentation with only a textual description the task without any image examples.
We explore the oulcomes of interchanging and combining image and lextual prompls in our model. We find that when using both modalities our model achieves improved ICL performance compared to past vision-only approaches (Figure 1), improving average precision by over 10% in Foreground Segmentation, over 4% for single object detection, and closing over 40% of the gap between current ICL approaches to state-of-the-art 1-shot, training approaches that utilize supervised base-class training data. Beyond visual recognition, TMProv can be applied to general vision tasks including edge estimation, depth estimation, and conditional image sest as shown in Figure 1.
2 Prompting Inpainting Models via Images and Text
We start by presenting our IMProv model and how to train it in Section 2.1, and subsequently, in Section 2.2, explain the approach for prompting the model using visual and textual prompts. Finally, in Section 2.3, describe the new dataset of images and associated captions collected.
2.1 IMProv Text Conditional Inpainting Model
We introduce a model with Inpainting-bast basedd Multimodal Prompting capabilities for vision tasks (IMProv). It receives both text and masked input image context and outputs reconstructed image.
Given an input image E IR Hx binary mask E {0, H) and sentence i E K x V where V is the vocabulary and K in the sentence length, the goal of our inpainting model f is to generate new image y E "xxx33 with the masked regions filled according to the input image context and the sentence:
= f(x, m,
(1)
Our model has encoder-decoder structure like MAE- VQGAN (Bar et al., 2022), where the encoder and decoder are Vision Transformers (Dosovitskiy et al., 2020). In contrast to Bar et al. (2022); He et al. (2021), after every sell-altention layer, we add a cross-at tention layer between image tokens and textual tokens, thereby effectively allowing each image token to attend to text token:
n
exp(K3 (KF)))))
2.
Σ ai Vi j=1
0ij m= exp(Km, Q.))
(2)
3

---

Published in Transactions on Machine Learning Research (09/2024)
Where V is the set of textual token values, K is the set of text token keys and Q is the set of image token queries. The resulting output sequence Z represents the attended image features that are most relevant to the text tokens.
Training, To train the model, the input image æ is split into patches and randomly masked by dropping a fixed percent of the patches (75% our experiments). Similarly, the input textual sentence tokenized and every token is mapped to its corresponding CLIP (Radford et al., 2021) embedding. Given the subset of non-masked patches and the textual tokens image, the model is then trained to predict the visual tokens corresponding to masked patches. The model is trained with pross-entrops loss applied between the model prodictions and the corrosponding visual tokens for cach masked patch. The ground truth visual tokens are obtained by mapping the imput image to visual tokens indices using a pre-trained VQGAN encoder (Esser et al., 2021). Formally, our text-conditioned MAE- VQGAN models the distribution p(z,|22, m,t), where 2, is visual token from the VQGAN vocabulary that corresponds to the i-th image patch.
2.2 Multimodal Prompt
At inference time, prompting the trained model can be done via text, via a visual prompt, or by combining both. To prompt the model via visual prompt we apply the same task formulation of Bar et al. (2022) form a grid-like image composed of task inpul-oulput example(s) (e.g. inpul images and their segmentation masks), and a new query image, and apply the model to inpaint the corresponding result for the query. To prompt the model via text, we provide to f a description of the task (e.g. 'Left: input images, Right: foreground/ /background segmentation results").
Table Examples of different textual prompts used for inference with IMProv.
Prompt
Full Text Examples
Text
Task
Image Segmentation
+ Location
Left input image, right: Black and white foreground background segmentation
Class Name
Left input image, right: Black and white foreground background segmentation a horse
Formally, Let S {(zi, u:) be the set of input-output examples where is an image and y; is the corresponding vision task output, let be a textual task description and let zy be a query image. We introduce an arrangement function 9i that arranges S and By into a grid (visual prompt), denoted by Top and provides a mask m for the inpainting function:
Top: m = 91 (5,
33)
Similarly, we have a corresponding arrangement function 92 that generates textual prompt that is used instruct the model how to inpaint the image given attributes like the task name, location details, and the image class name:
g2ttakk, loc, class)
(4)
For example, for the task of image segmentation of an airplanc, the output can be "Left: input image of an airplane, Right: corresponding image segmentation". For more examples see Table 1.
The model f then applied to reconstruct the masked area m given the visual prompt and the task description:
y = f m,
(5)
When the task description is an empty string, no textual prompt is given and the model has to infer the task solely from the examples of S. When S is empty set, and textual prompt is given, the model performs zero-shot completion by relying only on the textual instructions.
2.3 Image- Text Dataset for Computer Vision
4

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Dataset comparison.
The grid-like visual prompt images inpainted by IM- Prov have a different distribution from natural im- ages. Vision task descriptions (e.g. "left: input, right: segmentation mask'), paired with images, do not appear often in widely used language-and-vviion datasets. Thus, a model that was trained on these datasets will have trouble completing the inpainting task successfully due to the distribution shift. To mitigate this domain gap, we collect new dataset of figures, paired with their associated captions, extracted from computer vision papers.
Dataset
images
papers
with text
source
CVF
78,227
20,764
no
arxiv
S2CV
268,118
261,225
Yes
Semantic Scholar
Our dataset, The Semantic Scholar Computer Vision dataset (S2CV), is collected from computer vision papers that appear on "Semantic Scholar" website. This website contains papers from 40 conferences and journals from the years 2010 to 2022. We extracted pairs of figures and their captions from each paper on the website, resulting 1,417,398 pairs. We then filtered out figures that do not include images (e.g. plot of loss curve). Finally, the filtered S2CV dataset includes 268,118 captioned figures and is 3 times larger than the largest existing figures dataset, the Computer Vision Figures dataset (CVF; Bar et al. (2022)). See comparison in Table 2, full details about S2CV in the dataset datasheet (Gebru et al., 2021), and provided examples in Figure 11.
We also extend the existing CVF by repeating its data collection process and extracting the captions of the figures in the dataset. This results in 78,227 image-text pairs. This dataset, CCVF (Captioned-CVF) serves as baseline in our experiments.
3 Experiments and Results
We train a IMProv with a ViT-L backbone on a combination of our CCVF, S2CV dataset and LAION- 400M (Schuhmann et al., 2021). During the training process, we create mini-batches by randomly selecting half the data from the LAION-400M dataset and the other half from CCVF and S2CV, ensuring that the model learns from a diverse set of figuro-like images.
We evaluate TMProv on variety of computer vision tasks. By default, our visual prompt consists of 2 2 grid where the bottom-right quarter is masked, the top row contains the input-output example, and the bottom-left image represents the query. The visual example and the textual prompt are defined according to the task (see Section 3.2).
3.1 Implementation Details
During training, utilize images and their associated captions. We follow the resized cropping and flipping augmentations of He et al. (2021) and train on 224 x 224 crops. We use AdamW (Toshchilov & Hutter, 2017) optimizer with a learning rate of 2e and weight decay of 0.05. We train our models on one machine with 8 A100 GPUs with a batch size of 2048 for 150k iterations. Our learning-rato schedule consists of 2k lincar warm-up up steps followed by a cosine learning rate decay. We use a pre-trained frozen CLIP VIT-L/14 model as our text. encoder and a pre-trained VQGAN codebook with a vocabulary size of 1024, provided by Esser et al. (2021) and spatial dimension reduction of x 16. During training, we drop the text conditioning with probability of 0.1.
3.2 Downstream Computer Vision Tasks
Next, we include the evaluation results of IMProv on wide range of computer vision tasks. When trained CCVF/ S2CV and LAION 400M (Schuhmann et al., 2021), IMProv significantly improves ICL performance over wide range of computer vision downstream tasks when compared to vision-only ICL approaches.
Foreground Segmentation. In this lask, the goal is to segment the query image inlo two classes foreground and background. The onput-output example is a random image with its corresponding binary segmentation mask (e.g. black for the background and white for the foreground). We define the textual prompt to be: "Left segmentation of (class}", where the (class) is
input image, right Black and white the class of the foreground object, annotated in Pascal-5'. We follow the evaluation protocol of Bar et al.
5

---

Published in Transactions on Machine Learning Research (09/2024)
Image Segmentation
Segmentation Image
Exarple prorpt:
inage, right
segnentacion"
Example praspt: "Right output inage Left image, Left Sensatic tegmstation right inage"
Image > Edges
Edges + Image
Example pronot: "Left Input image, right
Edge map of [fnage description)"
prompt:: "Rrihht ourpuut image image, edge nap right inage"
Image > Depth
Depth Image
Exarple prorpt:
*Laft 1nout tmage, right Dopth naр of left image"
Exarpte
Left fmago,
left
Image > Normals
Normals
Image
Exaeple prompt: innut image, right
image*
Exarple prompt: "Right output image of left trage,
1eft. Nornal nap of right Tmago"
Figure& Multimodal prompting prediction examples. An example text prompt that was provided to IMProv together with the presented visual prompt appears below them. For each prompt, the result is marked in red. Please see the supplementary material for more results.
(2022) and test IMProv four splits of Pascal-5 dataset (Shaban et al., 2017). Results reported in Table 3.
Object Detection. Similar to the task of Foreground Segmentation, our objective here is to perform binary segmentation of the object present in the query image. However, this task is more challenging as the input-output examples contain a rectangle-shaped mask, derived from a bounding box which is less accurate compared to fine dctailed segmentation mask. We define the textual prompt be: "Left input image, right- Black and white foreground background segmentation of (class) of rectangle shape" where the (class) is the class of the foreground object. We use the Pascal VOC 2012 dataset (Everingham al., 2015), which

---

Published in Transactions on Machine Learning Research (09/2024)
Task Location
Task Location class
No Text
Task
Task + Location
Task Location class
ichils tangoound ausiground
iftile longround besiground sog ingmrmation wawsunnnaaaasnaad muu_nam
fomground backggmund regmantation
image sog sogmentation
tmagee pogmentatinn Klas, samel"
mmgmrtation
Figure 4: Detailed textual prompts IMProv performance. We experiment with textual prompts with varied amounts of detail, e.g.; from text to instructions that include the task, specific locations, and object class names. See examples of full-text prompts in Table 1. Please see the supplementary material for more results.
Table 3: Comparison between previous visual prompting results to multimodal prompting on computer vision tasks. For Foreground Sogmentation and Single Object Detection, we report the mIOU score. For Colorization, we report the MSE and LPIPS. Training dataset appears parentheses.
Model
Foreground Segmentation
Single Object Detection t
Colorization
Split 0
Split:
Split 2
Split:
Split:
Split
Split 3
Split
MSE
LPIPS
BEIT (CVF)
5.38
3.94
3.20
3.29
0.17
0.02
0.14
0.16
0.60
0.70
VQGAN (CVF)
12.56
17.51
14.27
15.06
2.27
2.37
2.48
1.99
1.50
0.56
MAE (CVF)
17.42
25.70
18.64
16.53
5.49
4.98
5.21
5.81
0.43
0.55
MAE VQGAN (CVF)
27.83
30.44
26,15
24.25
24.19
25.20
25.36
25.23
0.67
0.40
IMProv (S2CV + LAION)
42.58
44.81
40.73
33.72
30.03
30.73
29.8
31.32
0.57
0.34
consists of images along with their associated detection boxes. Our results are reported Table terms of the mean Intersection over Union (mIOU) metric.
Colorization. The goal is to map gray-scale image to color image. The example pair is gray-scaled image and the corresponding color image. We define the textual prompt to be: "Colorization results: Left input image, Right Colorized image of class" where the {class} is the class of object present in the image. We randomly sampled 1000 example pairs and image query from ImageNet (Russakovsky et al., 2015) validation set and converted them to gcay-scale to obtain gray-scale and color version for each image. MSE and LPIPS (Zhang et al., 2018) Results are reported Table 3.
Other Tasks. We evaluate our models the dataset created by Wang al. (2023c), which includes around 310k image-caption pairs that were automatically annotated by using state-of-the-an pre-trained models for wide range of vision tasks. Specifically, each image is annotated with depth and normal maps obtained from Midas (Ranftl et al., 2022). segmentation maps obtained from Uniformer (Li et al., 2022), and object boundary maps detected by HED (Xie & Tu, 2015). For each vision task X, we evaluate our model on two tasks X-to-images and images-tor to-X. As each task has different evaluation metric, and as our model produces image outputs, we simplify the evaluation by comparing the generated image to the renderod annotation of the task by calculating LPIPS (Zhang al., 2018). We report the results in Table and plot qualitative results in Figure 3.

---

Published in Transactions on Machine Learning Research (09/2024)
A
A
A
A
A
A
A
A
SD1
B
H
B
B
B
I
B
P
-
A
A
A
A
A
A
A
A
SD2
B
C
B
B
B
E
B
E
A
A
A
A
A
A
A
A
IMProv
B
8
B
B
B
E
B
T
Left imput letter image, right Red letter image lltter "B"
Left imput letter image, right Red letter image of letter "F"
Left input letter image, right Red letter image of letter "T""
No text prompt
Figure 5: Textual prompts effect on text inpainting. When there is inconsistency between the textual and visual prompt, the model may follow the textual prompt. IMProvean better follow both visual and text instructions compared to Stable Diffusion (SD) on letter generation.
4 Analysis
Dataset Ablation. We report our results and compare them with prior works in Table TMProv trained on a combination of LAION-400M and our S2CV dataset outperforms the prior work (Bar et al., 2022) trained solely the CVF dataset by more than ~ 12 points in mTQU. It demonstrates IMProv could benefit from training on additional amounts of unlabeled images.
Text helps. Adding textual prompts "Random Class" visual prompts improves Foreground Segmentation.
Table
Model
Avg.
MAF- (CVF)
23.52
Textual Prompts Ablation. We experiment with textual and visual prompts that have different relevance to the query image and task. For the visual prompt, we choose the input-output examples using three different retrieval strategies: (1) Random
IMPrOv(CCVE)
26.13
IMProv(CCVF + LAION)
36.29
Class, a random example pair in which the class of the foreground object is chosen randomly, (2) Same Class, where random example pair of the same class is chosen randomly, and (3) The example is chosen via Nearest Neighbor from all the images with the same foreground object class, using the model from Zhang et al. (2023). We evaluate our IMProv model on Pascal 5' with and without a textual prompt that contains Task, Location, and Class Name.
Firstly, we compare against Bar al. (2022) under (1) Random Class visual prompt setting and report results in Table 4. In this setting, the visual prompts describe the task (e.g., segmentation) but are not.
8

---

Published in Transactions on Machine Learning Research (09/2024)
Table 5: Training on additional annotated data improves ICL. We train IMProv on unstructured and structured data. We find that adding structured and fully annotated data during training can help ICL performance on 8 held-out computer vision tasks.
Depth > Image-
HED +
Image HED
Seg Image
Image Seg
Normals Image
Image >
Image
Depth
Image
Normals
Supervised TCL (InstrictPix2Pix) IMProv (S2CV+LAION) IMProv NOY+LLAIO-InstruetPPP
0.65
0.60
0.59
0.62
0.64
0.61
0.62
0.55
0.61
0,52
0.51
0.46
0.59
0.50
0.56
0.48
0.55
0.43
0.47
0.37
0.54
0.46
0.51
0.44
curated from the same class (the setting in Table 3), or chosen via nearest neighbors as in Zhang et al. (2023). Using non-curated visual prompts is most realistic, as finding perfectly aligned visual example might be hard as solving the original imput. The result shows that conditioning text improves average mloU by 3 points when using reasonable non-curated visual prompts. Moreover, IMProv trained on a combination of LAION and our CCVF dataset further boost the mloU by 10 points.
In Figure we plot the results under different textual prompts. We find that the textual prompts play a big role the perfor- mance of the model (see Figure 6). To dive deeper into the effect of textual prompt, we plot the relation between textual prompt and visual prompt in Figure 4. It shows adding text prompts improves the results for any type of visual prompt, from the least related Random Class examples to the most relevant Near- est Neighbors examples. In addition, we find that by using text, it is possible to achieve similar performance with lower-quality visual examples (using the Same Class example rather than the Nearest Neighbor (Zhang et al., 2023)). Similarly, higher- quality visual examples improve the results for all the tested textual prompts. Interestingly, the results suggest trade-off between the two modalities- high-quality textual prompls can alleviate the need for carefully chosen visual prompts, and vice versa.
42
38
Same Performance
mIOU (86)
36
+8%
34
32
30
28
W/o Text Prompt
Prompt
26
Random Class
Same Class
Nearest Neighbor
Visual Input: Output Example Relevance
Moreover, as shown in Figure 5, when the textual prompt is inconsistent with the visual example, the model may follow the more certain textual instruction. We include additional results for different combinations of visual and textual prompts in the Supplementary Material.
Figure 6: Using textual prompts improves performance and reduces the need for careful selection of visual prompts.
Does Structured Data Improve In- In-Context Learning?
The key insight of our approach to train IMProv on unstructured data a fully unsupervised manner without parsing the image or the text. Here we experiment with training IMProv on additional structured data in fully supervised manner. We use the dataset of Brooks et al. (2022), which consists of 310k onput-output image editing pairs and their corresponding descriptions. For training, as our onput-output examples. We embed them into a grid structure, in a similar manner to the structure we use at test-time. The grid images that we construct for training consist of 1 x 2 and 2 x 2 images by randomly selecting 1 or 2 input-output examples for each caption in the original dataset.
use random pairs
We test our models on a held-out set of vision tasks. As shown Table 5, we find that training on structurod supervised data alone leads poor generalization and ICL performance on the test lasks. Training on both unstructured S2CV and LAION-400M, together with the structured data improves ICL results the test tasks compared to our base model.
Comparison to Finetuning and Few-Shot Baselines. We compare IMProv to classic 1-shot. baselines, which we view as an upper bound of our approach. Approaches like FWB (Nguyen & Todorovie, 2019) and CyCTR (Zhang et al., 2021) utilize a fully labeled base classes train set (2086 to 5883 on different Pascal 5' splits) with architectures that are optimized for foreground segmentation (e.g, by utilizing higher resolutions). We also compare to MAR-VQGAN (Bar et al., 2022) that performs visual prompting without text and to
9

---

Published in Transactions on Machine Learning Research (09/2024)
SD1
SD2
IMProv
Figure 7: Compare to Stable Diffusion (SD) on Segmentation. The result marked in red.
finetuning baselines with K [1,4,,10 16) training examples for each target class. The results in Table 6 indicate that IMProv closes 40% of the accuracy gap between MAE-VQGAN to supervised one-shot approaches. This demonstrates the potential of our approach to scale with more data.
Comparison to existing text-to-image works. We also compare our IMProv against stale-of-the-arl text-image generative models, ie. Stable Diffusson Rombach et al. (2021). We input the same text prompt and visual prompt to Stable Diffusion version (SD1) and 2 (SD2) inpainting models, with inference steps and 7.5 classifier free guidance seale. The quantitative results are shown in Figure Compared to IMProv, SD fails to generate black/ white segmentation mask, and couldn't leverage visual prompt to
10

---

Published in Transactions on Machine Learning Research (09/2024)
Painter
IMProv
Figure 8: Comparison with other visual prompt methods. We compare IMProv with Painter Wang et al. (2023a) the task colorization. Both models are not trained explicitly with colorization supervision. Wang et al. (2023a) fails in some images, generating segmentation masks instead, while ours could generate reasonable colorized images.
MAE-VQGAN
IMProv
Task ambiguity
Non-aligned input-output
Out of dist. decoding
Figure Failure case analysis. Compare Bar et al. (2022), IMProv could succeed on some cases extend e.g. 'Task ambiguity". But still fail of the 'Non-aligned input-output' cases.
find the corresponding objects. We also compare them on the letter generation task in Figure 5. Although SD sometimes could generate the correct letter, it still fails to follow the visual prompt to generate white background and red font color. Moreover, as shown in Figure 5, when the textual prompt is inconsistent with the visual example, the model may follow the more certain textual instruction.
Compare to supervised approach. Compared a supervised prompting approach like Painter(Wang al., 2023a), IMProv can generalize to larger variety of tasks. We demonstrate this in Figure 8, showing that while Painter fails to adapt to the colorization task, IMProv performs reasonably well. We additionally report the comparison of Painter and our method in Table 7. Our model outperforms Painter by a large margin. It's also worth noting that changing the training datasct from CCVF to S2CV improves the mIoU significantly, which further indicates the effectiveness of our proposed S2CV dataset.
Table 7: Segmentation results on the FSS-1000 dataset.
Model
FSS-1000 mIoU
VisualPrompt Painter
58.3
62.3
TMProv (CCVF)
62.8
IMProv (S2CV)
68.9
11

---

Published in Transactions on Machine Learning Research (09/2024)
Table 6: Comparison to fine tuning and classic 1-shot segmentation baselines. MAE-VQGA image query and output resolution is 111x111. CyCTR and FWB resolution is 473x473 and 512x512, both approach utilize Pascal 5' labeled baseclasses data.
Pretraining
# Labelod Imnges
# Shots
Model
Split 0
Split
Split 2
Split 3
1
11.1
13.4
13.0
12.3
Unlabeled ImageNet
4
Finetumed MAE (Hl et al., 2021)
12.9
15.8
14.3
15.0
16
13.7
16.1
16.8
17.1
CVF + IN CCVF LAION S2CV LAION
1
MAE-VQGAN (Bar et al,, 2022)
32.5
33.8
32.7
27.2
1
1
IMProv
45.6
46.6
45.3
39.1
1
1
IMProv
49.1
49.7
45.5
42.1
Labeled Pascal (Segmentation masks)
FWB (Nguyen & Todorovic, 2019)
51.3
64.5
56.7
52.2
2086
5883
1
CyCTR (Zhang al., 2021)
67.2
71.1
57.6
59.0
Grid 3x3
Grid 4x4
Figure 10: Different grid size. IMProv also support different grid size other than 2x2 grid. We show segmentation results with 3x3 grid and 4x4 grid with different number of visual prompls. Note, the white region are discarded and not input to the model.
Table 8: Diffcrent grid size.
Different grid size With a 2x2 grid, only a visual prompt applicable. We also report different grid sizes with different numbers of visual prompts. We report the results on Pascal VOC segmentation in Table 8 and Figure 10. In each row, keeping grid size fixed, as we increase the number of visual prompt examples, the mIoU increases. On the other hand, due to the limitation of 224x224 resolution, when the grid size is larger, image in each shot becomes smaller, so one-shot accuracy drops. We acknowledge that the current model could only take a fixed image size, as many other pre-train image encoders. However, if more computation budgets are available, this issue could be mitigated with higher training resolution.
number of visual prompt
3
5
Grid 2x2
42.68
Grid 3x3
37,21
39.78
Grid 4xd
16.42
18.34
30.78
32.56
33.11
Failure case analysis We compare with failure cases of Bar et al. (2022) in Figure 9. For some of the cases, our IMProv could successfully address them with text prompts, e.g. the cat ion and bottle segmentation of "Task ambiguity". And our model could successfully generate the image that moves the orange to the center, though fails the other Non-aligned input-output example.
5 Related Work
Prompting in NLP. The ability to prompt a language model to solve a specific task, also known as ICL, a recently discovered property of generative language models that were trained on large corpus of text (Brown et al., 2020; Touvron et al., 2023; Chowdhery al., 2022; Bubeck et al., 2023). Brown et al. (2020) have shown that existing NLP tasks can be described in unstructured text, and then fed into large language model to complete the missing part without any finetuning (Radford et al., 2019; Brown et al., 2020). More recently different approaches to prompting have emerged including Prompt Engineering (Brown et al., 2020; Lu et al., 2021), Prompt Ensembllin (Jiang et al., 2020), Prompt Prefix Tuning (Li & Liang, 2021; Lester et al., 2021), and Chain of Thought Prompting (Wei et al., 2022). The Flamingo (Alayrac et al., 2022) model extends language-only models, and conditions the model on both image and text. Our approach different. in that our model outputs pixels and not text. Therefore, it is suited solve a variety of computer vision tasks that be represented in pixel-space, like semantic segmentation image colorization.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Visual Prompting. Recently, multiple papers have proposed methods to visually prompt computer vision models (Bahng et al., 2022; Jia et al., 2022; Bar et al., 2022). Bahng et al. (2022) proposes to add noise tensor to the input image to adapt the model to different tasks, while Jia et al. (2022) proposes to append learned tokens to Vision Transformers (Dosovitskiy et al., 2020), which draws motivation from prefix tuning in NLP (Li & Liang, 2021). These two approaches are trained on supervised data and thus struggle to scale and generalize to new tasks. Bar et al. (2022) takes a different approach and trains on unstructured crops from computer vision paper figures. According to this approach, visual prompting is viewed as an Image Inpainting task by creating image grid containing input-output examples and new image input. The goal of the inpainting model is then to complete the output in way that is consistent with the input. We follow a similar definition of visual prompt as in (Bar et al., 2022), however, we propose to condition the model on text ual input as well which might help to solve ambiguities in the task description and can more efficiently guide the visual model toward performing the desired task.
Image Inpainting and Image Synthesis. Early image inpainting methods relied on the input image itself for inpainting (Efros & Leung, 1999; Bertalmio et al., 2000; Criminisi et al., 2004; Barnes et al., 2009), whereas more recent works leveraged image datasets for this purpose (Hays & Efros, 2007; Pathak et al., 2016; Yang et al., 2017; Liu et al., 201 2018b;a). Lately, diffusion models have demonstrated large success in image inpainting and image synthesis (Ramesh et al., 2022; Rombach et al., 2021), as well as other popular transformer-based methods (Chen et al., 2020; Yu et al., 2021b; Esser et al., 2021; Yu et al., 2021a; Chang et al., 2022). Few of these approaches rely on discrete latent codebooks which induce a distribution over possible completions (Van Den Oord et al., 2017; Ramesh et al., 2021; Esser al., 2021; Yu al., 2021a; Chang et al., 2022). For instance, Esser et al. (2021); Yu et al. (2021a) proposed to synthesize images using autoregrossive model on codebook reprosentation, while Chang et al. (2022) applicd iterative parallel decoding of the tokens. Few approaches also support image synthesis with text conditioning MUSE (Chang et al., 2023), for example, is a transformer-base based model that applies cross-attention from image embeddings (VQGAN (Esser et al., 2021)) to the text embeddings extracted from a pre-trained model (c.g. T5 (Raffel et al., 2020)) to condition on text. Our model is conceptually similar to MUSE (Chang et al., 2023), however, we focus on inpainting grid-like visual prompts that require reasoning across multiple sub-images and input. text.
Few-Shot Learning. In this setting, the algorithm is trained on a labeled dataset of base classes, from which it should transfer to a set of novel classes given only few training examples (like 10 or 30) (Nguyen & Todorovic, 2019; Kang et al., 2019; Liu et al., 2020; Wang et al., 2020; Yang et al., 2020; Tian et al., 2020; Zhang et al., 2021; Bar et al., 2021). Unlike Few-Shot approaches, here we do not assume access to a large training set of base-classes, and our architecture is not task-specific. Our approach is Few-Shot only in the sense that the visual part of our prompt usually contains one or two task examples.
6 Discussion
We presented an approach for multimodal prompting of inpainting models. To unlock in-context learning capabilities in such models, we had to collect a specific dataset of figures with associated captions. To further scale this approach, we believe that other sources of unstructured data visual, textual, and multimodal should be incorporated during the training phase. To understand the feasibility of such data collection effort, one must be able to predict and quantify the effect of different dataset sizes and types on the downstream in-context learning capabilities of models trained on them. We plan to investigate it in future work.
13

---

Published in Transactions on Machine Learning Research (09/2024)
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynokds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 15-23716-23736, 2022. 12
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint (rXXx:220.11777, 1(3)44, 2022. 13
Amir Bar, Xin Wang, Vadim Kantorov, Colorado Reod, Roci Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Detreg: Unsupervised pretraining with region priors for object detection. arXiv preprint ar: 04550, 2021. 13
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005 25017, 2022. 2, 3, 4, 5, 8, 9, 11, 12, 13, 20
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference Computer graphics and interactive techniques, pp. 417-424, 2000. 13
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint 2Xxi:22111008000 2022. 9
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/ 201e/2005.14165, 2020. URL https://arxiv. org/ argg/bbs/005.... 14165. 1, 12
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat. Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Farly experiments with gpt-4, arXiv preprint 2rX1v:2303.12712, 2023. 12
Justine Cassell, David McNeill, and Karl-Erik McCullough. Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information. Pragmalics & cognilion, 7(1):1 34, 1999. 2
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35:18878- 18891, 2022.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. arXiv preprint arXiv:2202. 04200, 2022. 13
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- King-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-inage generation via masked generative transformers, 2023. 13
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Nya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691-1703. PMLR, 2020. 13
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Xiv preprint arXix:2204.0050 2022. 1, 12
14

---

Published in Transactions on Machine Learning Research (09/2024)
Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Region filling and object removal by exemplar-based image inpainting. IEEF Transaclions on image processing, 13(9): (099):1200-2122 2004. 13
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition scale. Xiv preprint 1xxx:2010.11922 2020. 3, 13
Alexci A. Efros and Thomas K. Leung. Texture synthesis by non-parametric sampling. In IEEE International Conference on Compuler Vision, pp. 1033 1038, Corfu, Greece, September 1999. 13
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. Tn Proceedings of the /EFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873-12833 2021. 3, 5, 13
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98 136, January 2015. 6
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford, Datasheets for datasets. Communiccaions of the ACM, 64(12):86 92, 2021. 5
James Hays and Alexei A Efros. Scene completion using millions of photographs. ACM Transactions on Graphics (SIGGRAPH 2007), 26(3), 2007. 13
Kaiming Hc, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autocncoders are scalable vision learners. CoRR, 0bs/2111.0637,, 2021. URL https: 5, 12
orggabs//111..... 06377. 3,
Menglin Jia, Lumnng Tang, Bor-Chur Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviu, Israel, Oclober 23 27, 2022, Proceedings, Part XXXIII, pp. 709 727. Springer, 2022. 13
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Associalion for Computational Linguislics, 8:423 438, 2020. 12
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via. feature reweighting. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 8420 8429, 2019. 13
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter--fficien prompt tuning. CoRR, abs/2104.08623 2021. URL https://arzis hwwp..or///aaae.. hts//bbb/2004.. /tt///211009990 08691. 12
Kunchang Li, Yali Wang. Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. CoRR, abs/2201 2es//201044666, 2022. URL 04676. 7
https:///rxivv httpa////aaxii.oooabba//////////
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint
arXiv:2101.0011 2021. 12, 13
Guilin Liu, Fitsum A. Roda, Kevin J. Shih, Ting- Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018a.
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European conference on compuler vision (ECCV), pp. 85 100, 2018b. 13
Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-aware prototype network for sho-shot semantic segmentation. In Furopean Conference on Computer Vision, pp. 142-158 Springer, 2020. 13
15

---

Published in Transactions on Machine Learning Research (09/2024)
Ila Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint orXi::1111.00100 2017. 5
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Pantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXivv:1144008766 2021. 12
David McNeill. Gesture and thought. University of Chicago press, 2019. 2
Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 622 631, 2019. 9, 12, 13
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016. 13
Aloc Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nyy Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. 12
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of mlr.
Machine Learning Research, pp. 8748 8763. PMLR, 18 24 Jul 2021. URL
(osss/v139/radforddoe html. 3, 4, 18
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2020. 13
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tlya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831. PMLR, 2021. 13
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. URL http:s:/aaxwvvoor//aaa/ /2d.///2.....2 /220/22222........ 13
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Poltern Analysis and Machine Intelligence, 44(3), 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021. 10, 13
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-255 2015. doi: 10. 10.1007//11233-015--66
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of fip-filtered 400 million image-text pairs. arXiv preprint arxiv:211 10t:2111.021111 14, 2021. 2, 5
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022. 18
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. CoRR, abs/ abs/1109.03410, 2017. URL http: //tpivv.///ii..e/a wrg//bs///09./.o////// 03410. 6
16

---

Published in Transactions on Machine Learning Research (09/2024)
Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pollern analysis and machine intelligence, 2020. 13
Hugo Touvron, Thibaut Lavril, Gantier Tzacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprind Xrx:22330. 199771 2023. 1, 12
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances neural informalion processing systems, 30, 2017. 13
Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. arXiv preprint arXiv:2003.0695 2020. 13
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist. painter for in-context visual learning, arXiv preprint Xr::::2212.00. 02499, 2022. 2
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion, pp. 6830 6839, 2023a. 11
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304, 03284, 2023b.
Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang Wang, and Mingyuan Zhou. In-context learning unlocked for diffusion models, 2023c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXim:2201. 210020991190,, 2022. 12
Saining Xie and Zhuowen Tu. Holistically-nesee edge detection. CoRR, abs/1504.06335,, 2015. URL http://arxiv. org/ aass///4. 06375.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:211.00000 2021. 1
Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In European Conference on Computer Vision, pp. 763-778. Springer, 2020. 13
Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6721-672 6729, 2017. 13
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:2110. 21X.:22110004227, 2021a. 13
Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. Diverse image inpainting with bidirectional and autoregressive transformers. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 69 69-78, 2021b. 13
Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems, 34, 2021. 9, 12, 13
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 586 595, 2018. 7
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context lcarning?, 2023. 8, 9, 18, 20
17

---

Published in Transactions on Machine Learning Research (09/2024)
Appendix
We include more information about the experimental study, as well as the Captioned Computer Vision Figures (CCVF) dataset datasheet.
A Broader Impact Statement
Using In-Context Learning to solve various vision lasks using one model has the polential to reduce the cost. of training and inference of future models, facilitate access to this technology, and democratize its exploration. Nevertheless, while scaling the training dataset to large corpus of non-curated text-image pairs improves the in-context learning results, it might also introduce potential biases that must be mitigated prior any commercial deployment of the model. A comprehensive discussion about ethical considerations be found in Schuhmann et al. (2022).
B Experiments and Results
We train IMProv CCVF/S2CV and LAION 400M and evaluate in-context learning. We first visualize the S2CV shown in Figure 11. We then explain the details of our experiments as follows.
Multimodal Prompting. Inspired by Zhang et al. (2023), we experiment with prompts that have different relevance to the input query image, we include visualization examples Figure 12.
For the visual prompt, we use five different retrieval strategies for choosing input-output example pairs from Pascal-5':
No input-output visual example No Example")
A random input-output pair in which the class of the foreground object is different from the class of the foroground object (* Differrent Class Random Sample")
A random input-output pair with the same foreground object class as in the query image (* Same Class Random Sample")
The nearest neighbor input-output pair in CLIP embedding space Radford et al. (2021) retrieved from all the images with the same foreground object class ("Same Class CLIP NN")
The nearest neighbor retrieved from all the images with the same foreground object class according to the model provided by Zhang al. (2023) ("Same Class Zhang et al. (2023) NN"). This strategy was trained to optimize in-context learning results.
For the text prompt, we experimented with three prompting variations:
No text prompt
Text prompt with location and lask information but without class "Left inpul image, right Black and white
segmentation"
Text prompl with location, task and class information "Left input image, right- Black and white srneground/backgrooong segmentation of {class]"
Table 10 and Figure present quantitative and qualitative results for different combinations of visual and textual prompts. We find that more informative textual prompts improve the results for all visual prompts. Similarly, higher-quality visual examples improve the results for all the tested textual prompts. Interestingly, the results suggest a trade-off between the two modalities high-quality textual prompts can alleviate the need for carefully chosen visual prompts, and vice versa. We argue that using non-curated visual prompts is most realistic, as finding a perfectly aligned visual example might be as hard as solving the original input.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Cantoc Semmeratio
Samentt Sww.ntesppanaete
WemSemarte Semerasion
Bascline
Ground Futh
CE Pred 25in
Fur
Grean Iruh
Bird
boat
dog
Figure 11: Image-
-nage-Text Text pair examples in S2CV.
19

---

Published in Transactions on Machine Learning Research (09/2024)
So in Table 9 we report the mloU on Pascal VOC segmentation, where we compare against Bar et al. (2022) under " "Different Class Random Sample" vi- sual prompt sctting. In this sctting, the visual prompls describe the Lask (e.g., segmentation) but are not curated from the same class (as Table 2), or chosen via nearest neighbors. The result shows that conditioning text significantly improves the prompting performance when using reasonable non- curated visual prompts.
Table 9: Improvement of textual prompt. Com- parison under " DDifferent Class Random Sample" visual prompting setting, IMProv outperforms MAE- VQGAAN with textual prompt.
Model
Split 0
Split 1
Split 2
Split
avg
MAEAN (CVF)
24.66
25.15
24.36
19.91
23.52
IMPrOv(CCVE)
26.15
27.38
29.37
26.13
IMProv(CCVF + LAION)
37.09
10.68
36.91
30.49
36.29
Table 10: Visual and Textual Prompts Combination. We evaluate our model on different combinations of visual prompts and textual prompts, with varying relations to the query images.
Visual Prompt Type
No Example
Different Class Random Sample
Some Class
Same Class CLIP NN
Same Class
Visual Prompt
Text Prompt
w/ class
Random Sample
Zhang et al. (2023) NN
/
18.39
/
18.75
26.73
31.25
38.14
39.07
/
34.50
36.62
41.30
42,17
/
36.29
39.33
41.99
42.68
Trade-off between the number of visual prompt and textual prompts We plot the mIoU w.r.t the number of support in Figure 13. We run the experiments under grid 4x4 setting in Table 8, with "Random Class" support image. Similar to Bar et al. (2022), as we increase the number of support visual examples, mIoU goes up. It is worth noting that when there are only or examples, the text prompt does not help. It is because the model couldn't generate meaningful segmentation with small number of visual supports due to the resolution (see Figure 10 for details). Under the setting that visual support number greater than 3, the text prompt consistently improves the results. implies that our text prompt orthogonal the number of visual support pairs.
Sittar Text
2
S.eeet
Figure 13: Trade-off between the number of visual prompt and textual prompts.
Other Vision Tasks. We provide more qualitative results for both Image-to-X task and X-to-Image task in Figure 14 and Figure 15 respectively, "X" can be any of the following: segmentation, edges, depth and normal.
Generalizability to new vision tasks Our model was never trained on specific vision tasks, and instead was trained on unstructured and non-annotated figures from computer vision papers. The data we train on is not constructed explicitly for specific tasks, and even if it is trained on some task-specific figures, it is usually not presented the way test model, as randomly crop the figures during training. Therefore, we believe the test task is generalized from different task combination instead of replicating from training lask. Similarly to(aar et al., 2022), our model can generalize to unseen diverse tasks as shown in Figure 16.
Datasct Ablation We additionally report the results of IM- (To(S2CV+LATON in the Table under the same "Random Class" " setting. We report the average mIoU over 4 splits. IM- Prov(S2CV + LAION) outperforms IMProv(CCVF + LAION) by 1.7 points, which justifies the effectiveness of our proposed S2CV dataset.
Table 11: Dataset ablation.
Model
Avg.
IMProv(CCVV + LAION)
36.29
(SPPov + LATON)
38.07
20

---

Published in Transactions on Machine Learning Research (09/2024)
Different Class Random Sample
Sample Class Random Sample
Sample Class CLIP NN
Sample Class Sup.
No Example
No
Text Prompt
Text Prompt w/o class
Text Prompt
w/ class name
No Text Prompt
Text Prompt class name
Text Prompt w/ class name
No Text Prompt
Text Prompt w/o class name
Text Prompt w/ class name
Figure 12: Visual and Textual Prompts Combination. Each column represents visual prompt of different relevance. The result is marked in red.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Image- > Segmentation
Image Edges
Image- Depth
Image
Normal
Figure 14: Image-to-X results. The result marked red.
22

---

Published in Transactions on Machine Learning Research (09/2024)
Segmentation
Image
Edges
Image
Atwii
Depth
Image
Normal
Image
Figure 15: X-to-Image results. The result marked red.
23

---

Published in Transactions on Machine Learning Research (09/2024)
Figure 16: More vision tasks. We show IMProv is capable of other vision tasks, including edge detection, debluring, image enhancing, style transfer, image-to-sketch.
24


Reviews:
[{"review_id": "cts7h8mOlh", "reviewer": "Reviewer_Qrwx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The proposed dataset is interesting.\n* The proposed approach shows promising results against prior work such as MAE-VQGAN, Stable Diffusion and classic 1-shot segmentation baselines. It also shows promising performance against prior visual prompting methods such as Painter.\n* The paper is well written and easy to understand.\n* The paper includes a variety of ablations.\n\nWeakness:\n* Novelty: The key differences of the proposed method IMProv to Painter (Wang et. al. 2023) is not clear. Painter also uses in-painting on a 2x2 grid. Both models are trained using masked image modelling.\n\n* Do the results in Table 3 use the nearest neighbor in-context examples? What happens when an appropriate nearest neighbor  in-context example cannot be found? It would be better to report the results both with random in-context and nearest neighbor in-context examples.\n\n* Comparison to Painter: In Table 3 it would be helpful to compare to Painter (Wang et. al. 2023), as the proposed method and Painter are both very similar.\n\n* Comparison to supervised SOTA: Table 3 should also include the best supervised SOTA approaches, as results in Fig. 3 confirm that the proposed approach displays very weak performance on a variety of tasks. This will confirm is the proposed approach is usable for real-world tasks,\n\n* Image to depth color bleeding: From Fig. 3 it appears that e.g. depth maps are incorrectly predicted -- are are color pixels instead of grayscale. How are such invalid cases handled? \n\n* Can the proposed approach be extended to (multi-class) semantic segmentation or is it constrained to binary back/foreground segmentation?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a approach to solve unseen computer vision tasks at test time. The contributions are two fold: the paper presents a new dataset of image-text pairs from Semantic Scholar; and the paper presents a method -- IMProv, to inpaint masked regions given\nthe rest of the image and a caption. The image here contains an in-context example in a 2x2 grid. The method is evaluated on various tasks such as foreground/background segmentation, edge and depth estimation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The paper should discuss it's novel aspects in more detail. In general, the introduction of the paper should be clearer including the \"teaser\" figure in Figure 1.\n* The paper should expand Table 3 to include the evaluations discussed above.", "broader_impact_concerns": "* The paper should include a Broader Impact Statement clarifying any ethical concerns regarding scraping data from Semantic Scholar.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CxB0prgBgQ", "reviewer": "Reviewer_47rH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) It's good to see that using only in the wild laion data and cv paper data and training from scratch instead of initialized from pre-trained generative models, the model can achieve good results\n2) Using VQ-GAN for generation offers better alignment and controlabiility than diffusion model as shown in figure 6 and 7\n3) Text prompt helps to solve some hard cases in MAE-VQGAN as shown in Figure 9\n\nWeaknesses:\n1) Like previous work, this type of methods are trained on one-shot (2*2 grids), which is hard to extend to long context in-context learning naturally, and the num of shots are also barriered by resolution.\n\nSummary:\nThe self supervised model's performance is still far from supervised ICL baseline. I acknowledge the contribution of this paper is good, but there is still a far way to achieve language model level in-context learning capability using in the wild data. I support this paper to be accepted, the contribution is good and the exploration is necessary. I appreciate the solid experiments in this paper. I hope the authors to make some changes and provide more experiments results, since it will provide benifitial insights into future research on this unsupervised vision in context learning area.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a multimodal version of Visual Prompting via Image Inpainting, which introduces a new multimodal prompted in-painting based computer vision, self supervised, in context learning framework. The prompt comes from two modalities: 1) visual prompt is coming from unmasked patches of the image 2) text prompt is coming from annotations (task, location, class names); The authors collect a new dataset from semantic scholar, and train a MAE-VQGAN model from scatch for this purpose. After training on collected cv paper images and laion images, the model can emerge some capabilities for various cv tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Better discuss the difference and relation with Visual Prompting via Image Inpainting\n2) Try more annotated supervised data like (SAM/depth anything/Grounded DINO/hed, etc) for Supervised ICL, I don't think instructpix2pix are suitable for depth/seg/hed/norm\n3) Is it possible to also fine-tune SD inpainting model with your data recipe. It would show the benefit using MAE-VQGAN\n4) Can authors provide some results like finetuning baseline in Table 6 with K=1,4,16 for IMProv, to see if it can scale to more shots", "broader_impact_concerns": "No ethical concerns .", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TfB6gx4h3b", "reviewer": "Reviewer_UUw9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper is clearly written and easy to understand.\n2. There are extensive ablation studies to compare different training factors. \n\n**Weakness**\n\n1. The main concern is that the comparisons are not fair. In Table 3, all the baselines are trained with CVF, while the proposed model is trained with a much larger training set (i.e., S2CV+LAION). The same is true for Table 6. \n\n2.  While authors show that providing examples is beneficial to the model inference, the vision encoder takes a fixed image size. This may lead to a degraded performance when increasing the number of visual prompts. Such issue may limit the practical usage of this proposed model.\n\n3. It is not clear about how to select in-context learning examples. Are they randomly selected or based on certain criteria?\n\n4. Compared to the most relevant baseline MAE-VQGAN, the main contribution seems to be adding additional captions can provide guidance during model inference (as shown in Table 4). This observation is straightforward that may make the argument of in-context learning weak.\n\n5. While authors provide analysis about S2CV and CCVF, it is not clear about how these types of datasets can help in-context learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a generative model that is able to in-context learn visual tasks from multimodal prompts. The inputs are a textual description of a visual task, a few input-output visual examples, or both, the model in-context learns to solve it for a new test input. \n\nExperiments show that training this proposed model with text conditioning and scaling the dataset size improves in-context learning for computer vision tasks by a large margin. Extensive experiments demonstrate that vision and language prompts are complementary.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In general, authors provide a simple yet effective approach to address how to apply in-context learning to vision tasks. \n\nThe collected dataset can contribute to the multimodal understanding community if it can be open sourced. \n\nBased on the above weakness, the concerns are about the unfair comparisons, limitations of this approach, and the main contribution.", "broader_impact_concerns": "The reviewer did not see any obvious ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the proposed dataset (Qrwx, 47rH), the achieved results (Qrwx, 47rH), the accompanying analyses (Qrwx, 47rH, UUw9), and the presentation (Qrwx, UUw9).

The main concerns raised by the reviewers were on the experimental comparisons, deemed potentially unfair due to the different pertaining datasets (UUw9), and the novelty and comparison with Painter (Wang et al., 2023, Qrwx), as well as concerns on the resolution and grid size (47rH, UUw9). The authors' response addressed most of the concerns, providing a comparison with Painter (now Table 11 of the revised manuscript), analyses on the grid size (Table 8), and clarifications on the comparisons. 

Two reviewers recommended acceptance of the manuscript, valuing the effectiveness of the approach and the experimental efforts (47rH, UUw9), while Qrwx shared concerns regarding the technical novelty, the issues on resolutions, and the fact that the dataset is deemed the core contribution. 

The AE finds that IMProv, while extending previous works, is still a valid contribution to the community (together with the proposed dataset) as its simplicity is an added value given that the topic is under-explored and complex baselines are not strictly necessary at this early stage. Thus, the AE recommends the acceptance of the work. In the final version, the authors are advised to perform some minor changes, i.e.:
- Given that the paper already discusses Painter and compares qualitatively IMProv with it (Fig. 8), Tab. 11 and the corresponding paragraph should be added to the main manuscript (fixing also the issue with the reference to the corresponding table).
- Include a discussion on the impact of the resolution and grid sizes within the main manuscript (following the discussion with 47rH and the limitation stressed in the reply to UUw9).
Decision: Accept (poster)