Visual Prompting via Image Inpainting
Authors: Amir Bar
Yossi Gandelsman
Trevor Darrell
Amir Globerson
Alexei A Efros
Venue: NeurIPS 2022 Accept (2022)
Keywords: Computer Vision
Self-Supervised Learning
Visual Prompting

Abstract: How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification? Inspired by prompting in NLP, this paper investigates visual prompting: given input-output image example(s) of a new task at test time and a new input image, the goal is to automatically produce the output image, consistent with the given examples. We show that posing this problem as simple image inpainting -- literally just filling in a hole in a concatenated visual prompt image -- turns out to be surprisingly effective, provided that the inpainting algorithm has been trained on the right data. We train masked auto-encoders on a new dataset that we curated -- 88k unlabeled figures from academic papers sources on Arxiv. We apply visual prompting to these pretrained models and demonstrate results on various downstream image-to-image tasks, including foreground segmentation, single object detection, colorization, edge detection, etc. Project page: https://yossigandelsman.github.io/visual_prompt

Visual Prompting via Image Inpainting
Amir Bar"
1.2, Yossi Gandelsman"
Trevor Darrell', Amir Globerson 2:3, Alexei A. Efros
UC Berkeley
Tel Aviv University
Google Research
Abstract
How does one adapt pre-trained visual model to novel downstream tasks without task-specific finetuning any model modification? Inspired by prompting in NLP, this paper investigates visual prompting: given input-output image example(s) of new task test time and a new input image, the goal is automatically produce the output image, consistent with the given examples. We show that posing this problem as simple image inpainting literally just filling in hole in concatenated visual prompt image turns out to be surprisingly effective, provided that the inpainting algorithm has been trained on the right data. We train masked auto-encoders on a new dataset that we curated 88k unlabeled figures from academic papers sources on Arxiv. We apply visual prompting these pretrained models and demonstrate results on various downstream image-to-image tasks, including foreground segmentation, single object detection, colorization, edge detection, etc
Introduction
In the past few years, self-supervised learning has gained popularity in computer vision and natural language processing (NLP). The growing capacity of modern deep learning models made them prone to overfitting when trained on relatively small labeled datasets. Self-supervised learning provides solution to this problem by generating "free labels" for any dataset, without the need for manual annotation, addressing the data hunger in these high-capacity deep learning models. However, features learned via self sulf-vuueriii given downstream task by fine-tuning on some labeled dataset. Could this fine-tuning be avoided?
not "ready for use" they typically need to be adapted for
In NLP, prompting [5] has recently emerged as way to employ model for new task without any additional training. A common way of task prompting for specific language understanding task at test time is to provide the trained model with an input corresponding to example(s) of the target task together with the query. E.g.. typing the following input prompt:
Je suis désolé J P'dore la glace
I'm m sorry
will prompt the model [5] to perform the task of French-to-English translation, returning:
I love ice cream
Can this idea of test-time task prompting be generalized the visual domain? That is, instead of the current situation in computer vision, where each trained model serves its predefined task (e.g. segmentation, detection, classification), can we have a single general model that can perform wide range of user-specified tasks without any fine-tuning (i.e., weight modification)?
In this paper take step toward this goal by demonstrating that large-capacity image inpainting models, when trained on the right data, can be surprisingly effective tools for visual prompting.
Equal contribution.
Project page: https://yossiiaaddel/e hww.rsiiiganelsaaainiiihbhbbbbbb it/iii/viiii soovissallproopt
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Xip
X
W
Concatenate into single image
Inpainting Model
Task Input Example
Task Output Example
Query
Visual prompt image
Output
Edge detection
Colorization
Inpainting
Segmentation
Style transfer
Figure 1: Visual prompting via Image Inpainting, Top: Prompting Image Inpainting Models. Given input- output example(s) (21,21) y1) and image query 2q. construct grid-like like single image called visual prompt Tup. The visual prompt composed of the desired task example(s) and new query image (all in green). The inpainting model goal is then to predict the masked region (red) such that consistent with example(s). Bottom: an inpainting model can solve this way various computer vision tasks, given that it was trained on the right data. The model predictions annotated in red.
As shown in Figure T we define each task by constructing a new grid-like image that contains an input-output example(s) of the task and novel query (green border). The input-output example describes the task, and the image query defines a new input. The model then produces the result by simply inpainting the rest of the image (red border). This setting is most similar the classic Image Analogies [22) formulation, but is less constrained: instead of explicitly defining the A, A', and B images separately, we simply concatenate them into single image with hole (hence, visual prompting is not exactly an analogy since there is no implied reft-to-right ordering). Our goal is also dissimilar from the aims of meta-learning and few-shot learning methods, except that we make no distinction between tasks and example pairs. The only requirement of our formulation is that the tasks must be defined image-to-ime -mae translations, which very large subset of vision problems.
To obtain training data that most useful for our framework, we utilize a domain that spans across a variety of computer vision tasks figures and infographics from computer vision articles available Arxiv. We build large dataset of 88 thousand figures, many of which contain grids of images and their corresponding task results (e.g. images and their segmentation masks/stylized versions/edges, etc.). We then train large-capacity inpainting models to predict randomly masked patches from figures given other patches from the same figure.
Our main contributions are as follows. First, we present a simple yet surprisingly powerful general approach for visual prompting. We show that various computer vision tasks can be treated as grid inpainting problems, given a few examples of task inputs and outputs and query image. Second, provide
dataset that allows model learn such grid structures without any labeling, task descriptions, or any additional information about the grid structure. Finally, we show that while using new dataset for training is essential, adding more generic image data from other sources (e.g. ImageNet) further improves the results.
Related Work
Natural Image Inpainting. Filling empty regions in an image has been widely explored for natural images. Earlier methods used data from the input image itself for inpainting [14
49 whereas 31. Recent methods have
later works utilized datasets of images as source data [19 38
56
30
Due to exponentially large number of completion options for single output patch, these approaches rely discrete latent codebook [51 40 15] which serves as a smaller yet expressive vocabulary. To tackle the multimodal nature of synthesis, different approaches have been proposed to model the distribution over possible
attempted to apply transformers to visual synthesis tasks
15
57

---

proposed to synthesize images line-by-ling by-liie using an autoregressive model and 7 have proposed iterative parallel decoding. While the standard inpainting task typically aims to complete blank parts in natural images, our focus is on completing grid-like visual prompts, which require reasoning across multiple images within the visual prompt image.
completions [15 57 7 For example, 57 |7
Hole-filling as a Pretext task. Recent work has shown that self-supervised pretraining can generate powerful representations for transfer learning, even outperforming its supervised counterparts on
challenging vision benchmarks 37 18 21 TO 35 [6]. Pathak et al. [38] first proposed using hole-filling as pretext task for self self-supprvision with Context Encoders, where the goal is to predict an random image region given its context. Based on the recent success of Vision hole-filling a self-supervised pretext task for ViTs I 20 14]. For example, in MAE 20), the goals is to reconstruct the image given a small subset of input patches. After pretraining on unlabeled data, MAE can produce representations that transfer well when fine-tuned on downstream tasks. Here, we use visual prompting to adapt these models to downstream tasks without any finetuning.
Transformers (ViTs) [13, multiple works have proposed
Few-Shot Learning. In this setting, the algorithm is trained on a labeled dataset of base classes, from which it should transfer to set of novel classes given only few training examples (like 10 or 30) [36 27 32 53 55 48, 59 2]. Unlike Few-Shot approaches, here we do not assume access to a large training set of base-classee and our architecture not task-specific. Our approach is Few-Shot only in the sense that we construct visual prompt that contains one or two task examples.
Image Analogies. Hertzmann al. 22 proposed the framework of Image Analogies for texture synthesis, where the algorithm is given a pair of training images (A and A') and a query image (B). The goal is synthesize new image (B') conditioned the query, following the relationship inferred from the training pair. Other works have used analogies in style transfer (50), and as a supervised image synthesis task [4]. Predicting the correct completion was previously modeled as a classification problem [43 23 and other works have explored analogies in the context of
learning different transformations between pairs of images 34 46 Unlike these approaches, we use inpainting MAE models that learn from data, without assuming any predefined analogies structure.
Prompting in NLP. With the recent success of large unsupervised language models 44 12], Brown et al. 5 presented how a variety of NLP problems can be reformulated to a text completion problem given predefined prompt, which can be used solve different tasks without any finetuning. Prompting was shown to be a useful tool for solving various NLP tasks and benchmarks [39 5
More recently different approaches to prompting have emerged including Prompt Engineering [5] 33)
Prompt Ensembling (25), and Prompt Prefix Tuning [29] 28 Inspired by the success of prompting in NLP, we aim to study prompting in computer vision where prompting hasn't been widely explored.
3 Visual Prompting via Image Inpainting
We turn describe how to perform visual prompting using Image Inpainting models. In Section we describe our proposed inpainting model, which is a combination of MAE and VQ- GAN. We then proceed to discuss visual prompt- ing and propose different ways to create visual prompts in Section (see example in Figure []. Finally, we describe the dataset collected for training our model in Section 3.3. The training process is illustrated in Figure
Figure 2: MAE VQGAN Architecture. During train- ing, an input image is patchified, masked and fed into an MAE (20). For each masked token, the decoder outputs distribution over pretrained VQGAN [15] codebook. The model trained using cross entropy loss.
8100 100,51,
Cross Entropy
VOGAN Encoder (Pre trained)
3.1 Inpainting using MAEAVQGAN
VIT Decoder
Given an input image r E R and bi- nary mask m E {0, 1] HIW" the goal of an in- painting function ff is to synthesize a new image y E IR xxx33 with the masked regions filled:
VIT Encader
y f(z, m)
Input image
Random (non-mesked) imput petches
3

---

To implement f with neural network, it is necessary to consider design choices like the network architecture, how train it, and whether it outputs distribution over possible completions or pixels. We propose the MAF-VQGAA Following the design of MAE, the model is based on ViT [22 13 and it is trained via masked auto-encoding by randomly masking image patches and then applying f to reconstruct the image from the non non-masked parts. MAE-VQGAN models the distribution pa (z, m), where 24 E V is a visual token from a VQGAN vocabulary V that corresponds the hth ViT patch. For simplicity, we use fixed ImageNet pretrained VQGAN codebook [ MAE-VQGAN assigns probabilities to visual tokens via softmax layer, which is better suited for capturing ambiguities. During training, we obtain ground truth visual tokens by mapping the image to visual tokens indices using the VQGAN encoder. The model is trained using cross entropy loss.
model, which combines ideas from MAE 2 and VQGAN [1S]
Unlike MAE which directly predicts pixels,
Let = (21,.., 21. be the ordered set of predicted visual tokens. To obtain &, we use argmax:
& arg max po (25/2,, m)
Then, to decode the visual tokens to pixels, we apply VQGAN decoder to 2 to obtain y.
3.2 Prompting Inpainting Models
To prompt an inpainting model, we construct a visual prompt, a grid-like image composed of task input-output example(s), and new query image. The model then has to inpaint the rest of the image such that it is consistent with the task defined in the examples (see Figure IT.
Let S {mr, y= be the set of input-output examples where I; is image and y/ is function of x; (e.g y is segmentation mask). We assume is small (one or few examples). Then, given S and new input query T.. the goal is predict the corresponding label yo. To prompt the inpainting model discussed in Section 3.1 we need to define a function that maps the examples set S and query image Fg to new image and a mask:
[Zeps m] x())
The image Top is the visual prompt and the mask m defines the masked region f has to predict. For a given task, there might exist multiple implementations of g that can be considered. The goal of the inpainting model is to reason about the visual prompt Tup, and output a plausible completion without performing any additional training:
Vup = I (Iup. m)
To obtain Vg. we just take the part of Vep corresponding to the mask m.
Visual Prompt Engineering. For the visual prompting to work, g should output a good visual prompt, composed of the examples S and query image Tyg. Therefore, has to determine where and how to embed the inputs in the visual prompt image, considering the nature of the completion task. All the functions g used in this work were hard-coded and manually engineered. In most cases, stacks the examples and image query horizontally by creating an image grid of (n + 1) x cells, where the :th example is placed in the 5th row, and the image query is in the last row. The grid has fixed size, and therefore before populating it the input-output example pair(s) and query are first resized. Another consideration is how to draw every (æt, y:) pair. For example, y is segmentation mask, we can choose to use different colors to draw it. In Section 4.4 we describe different prompt design choices and their effect on the results.
Visual Prompt Ensembling, There could be multiple options to define The idea in prompt ensem- bling, inspired by NLP [25 28), to construct multiple different prompts, apply the inpainting model I on each prompt individually to obtain a set of predictions. The final prediction can be determined, for example, via majority voting, or weighted average. For simplicity, here we use simple average.
3.3 The Computer Vision Figures Dataset
The images produced by g are by construction not natural. Specifically, these images have a grid-like figure structure that stitches together images coming from different distributions, like natural images and segmentation masks. Therefore, a model trained on standard dataset (e.g., ImageNet (2) ) might struggle to process these grid-like images. To mitigate the domain gap, collected new dataset.
Using a publicty available checkpoint from https:/ggiiubb.oom//onniiiii tranfoosssss

---

horse
R00
Figure Random images from our Computer Vision Figures dataset. We curated dataset unlabeled figures from Computer Vision academic papers. During training, we randomly sample crops from these ligures, without any additional parsing.
The Computer Vision Figures (Figures) dataset consists of 88, 645 images that more closely resemble the structure of our visual prompts. The dataset was collected from Arxiv, the open-access web archive for scholarly articles from variety of academic fields. Arxiv sources are publicly available to download starting from 2010. We downloaded all paper sources from 2010 to 2022 and selected the Computer- Vision partition (cs.CV" sources, as they contain images that more closely resemble To remove unrelated source images like graphs charts,
grid structure, shown in Figure we manually tagged 2000 images and trained a binary image classifier to assign a high score to source images in figure-like like structure with at Ieast one natural image. We then used the classifier over the entire data to keep only the most informative source images, coming from 23, 302 different papers. We randomly partitioned 90% of the data train and left the rest for validation. We include a datasheet with more information in the Supplementary Material.
Experiments and Results
To study visual prompting, we pretrain different models (see Section 4.1 on ImageNet and on the Figures dataset, then quantitatively evaluate the models using different prompts on simple downstream computer vision tasks (see Section Using a synthetic dataset, we assess how the choice of model and data affect the success of visual prompting in Section 4.3 and explore different prompting design choices in Section 4.4 We provide large variate of qualitative results both this section as well as in the Supplementary Material.
4.1 Models and Baselines
To study the effect of model choice on prompting results, we experiment using different models, in- cluding MAE-VQGAN (see Section 3.1 and several other inpainting models briefly described below.
VQGAN 15
is autoregressive transformer model used for inpainting and image generation. Visual tokens are predicted sequentially, line-by-line, and the model is trained using cross-entropy loss. The VQGAN model codebook used encode visual tokens, and it trained beforehand using perceptual loss [26] and GAN loss [17]. We train it on ImageNet and our Figures dataset, following hyperparams in [15]. and use a pretrained codebook with vocabulary of size [V| 1024.
BETT [ is a masked auto-encoder. The model maps each input 16 x 16 patch to a visual token from d-VAE [40] vocabulary of size 8192. To encode each visual token, the image is first resized to 112 x 112 and then mapped to 196 tokens. We use the publicly available BEiT large model, pretrained on ImageNet-21k.. We also pretrain large BEiT model on Figures for 1000 epochs.
MAE 20 Similar to BEiT, MAEs attempt to reconstruct a masked input image. Unlike in BETT, the model directly regresses pixels and it is trained with (2 loss. During pretraining, only mon-masked
5

---

Table 1: Visual prompting results on computer vision tasks. For Foreground Segmentation and Single Object Detection, we report the mIOU score. For Colorization, we report the MSE.
Model
Foreground
Segmentation
Single Object Detection
Colorization
Split 0
Sphi
Split 2
Split 3
Split
Split 2
Split 3
Split 4
MSE
LPIPS
Copy
12.92
17.90
13.52
15.29
12.14
13.50
13.03
12.38
2.63
0.75
BETT (IN-21K)
0.38
0.93
0.90
0.95
0.24
0.32
0.19
0.10
1.25
0.73
VOGAN (IN-IK)
6.96
10.55
9.59
9.43
5.19
4.99
5.09
5.10
2.44
0.66
MAE (IN-IK)
1.92
6,76
3.85
4.57
1.37
1.98
1.62
1.62
1.13
0.87
MAE- VAA-NQGAN (IN-1K)
2.22
7.07
5.48
6,28
3.34
3.21
2.80
2.80
3.31
0.75
BETT (Figures)
5,38
3,94
3,20
3.29
0.17
0.02
0,14
0.16
0.60
0.70
VQGAN (Figures)
12.56
17.51
14.27
15.06
2.27
2.37
2.48
1.99
1,50
0.56
MAE (Figures)
17.42
25.70
18.64
16.53
5.49
4.98
5.24
5.84
0.43
0.55
MAE-VQGR VQGAN (Figures)
27.83
30.44
26.15
24.25
24.19
25.20
25.36
25.23
0.67
0.40
tokens are fed into the encoder, which results in a faster training time. We use publicly released checkpoint pretrained on ImageNet, and pretrain another model for 1000 epochs our dataset.
Copy Example. This simple baseline simply replicates the first example label as the output.
Implementation Details. All the models we describe are large transformer-based models 522 13 with patch size 16x16, embedding dim 1024, 24 layers, and 16 heads. For training, we used machine with 8 Quadro RTX 6000 GPUs, with a batch size of 48. The input image size is 224 x 224.
4.2 Downstream Computer Vision Tasks
We quantitatively evaluate the inpainting models described above on computer vision tasks.
Visual Prompt. Given one example pair and a query image, we structure the prompt in the same fashion for all tasks. We construct grid of 2 sub-images, where the example pair is embedded in the first row, and the query image appears in the bottom left cell. See the example in Figure
Computer vision tasks. We evaluate the inpainting models on standard image to image tasks like Foreground Segmentation, Single Object Detection and Colorization.
Foreground Segmentation. The goal is binary-segment the query image Foreground and Background. The example is an image and corresponding binary segmentation mask. The query is new image, and the goal is complete a corresponding segmentation mask. We use the Pascal-5i [5 dataset, which is comprised of 4 different image splits where every split contains between 346 and 725 images and associated segmentation masks. For each class, the data contains a few image- mask pairs, together with held-out image queries. For every image query, we choose one random example pair. To evaluate, every pixel in the completed image is first mapped to the nearest Foreground or Background color. Finally, report the mean IOU (mIOU) metric.
Single Object Detection. Similarly to Foreground Segmentation, the goal here is to binary-segment the object that appears the query image. However, this task is more challenging than Foreground Segmentation because the example mask is obtained from bounding box which is more coarse than a segmentation mask. We use the Pascal VOC 2012 dataset using images and their associated detection boxes. For simplicity, object and filter out trivial images that have an object covering more than 50% of the image. We ran- domly select example pair and image query of the same object class and repeat the process with 4 different random seeds. For evaluation, we follow a similar process as in Foreground Segmentation obtain binary segmentation mask. Then we keep the connected component with the largest area using morphological operations and draw a bounding box around it. We report the mIOU results.
Pascal annotations to include only images with a single
Colorization. The goal is to map a gray-scale image to color image. The example pair is gray- scaled image and the corresponding color image, as shown in Fan We randomly sampled 1000 example pairs and image query from ImageNet 42) validation set and converted them to gray- scale obtain gray-scale and color version for each image. We report the MSE loss and LPIPS (60)
Results. We include quantitative results in Table 1 and qualitative completion results in Figure 4 Training on the Figures dataset improves the results for most models in all the downstream tasks. MAE-VQGAN outperforms the other models by large margin for detection and segmentation and

---

Segmentation
Colorization
Inpainting
Edge detection
Figure 4: Visual prompting prediction examples. Each visual prompt was fed MAE-VQGAN model trained on Figures dataset. For each visual prompt, the result marked in red.
Color change prediction
Shape change prediction
Resize prediction
Resize & color change prediction
Resize & shape change prediction
Color & shape change prediction
Figure Synthetic data study results. MAE-VQGAN predictions are annotated red square.
generates much sharper images than the MAE. We find that VQGAN struggles to output accurate results, likely due sequential decoding. The BEiT model is outperformed by MAE, most likely because its training process is less sample efficient. For more results, see the Supplementary Material.
4.3 Synthetic Data Study
To assess the compositional prediction capabilities of inpainting models, we created of 3 simple synthetic tasks and of their combinations, and evaluated each model on 100 examples per task.
Visual Prompt. Given two example pairs and query image, structure the prompt in the same fashion for tasks. We construct a grid x 2 sub-images, where the example pairs are embedded in the first two rows, and the query image in the bottom left cell. We include examples in Figure
Change prediction tasks. Each example pair is an image of colored shape, and corresponding image with an introduced change. The change can be either in color, shape, size combination of two changes. Next, we describe each individual task in more detail.
Resize. Each example pair contains image of circle, and corresponding image with the circle smaller in size. The goal to predict the image with the resized version given image query.
Shape. Here every example pair is an image with circle, and a corresponding image with rectangle. Both are similar in size and appear the same location. The goal is predict the image with rectangle, given a new image query.
Color. Each example pair contains an image of circle appearing in the same location, with the color changed from green to blue. Given image query, the goal is predict the corresponding image with the circle colored in blue.
Evaluation. We map each predicted pixel to its nearest neighbor color from a predefined set of options: black, white, blue, or green, We then measure and report the color-aware mIOU, by considering pixel predictions that appear in the ground-truth -outh shape color foreground and treat the rest as background.
7

---

Table 2: Synthetic data study results. We report the color-aware mIOU on the six tasks.
Color
Shape
Size
Color & Shape
Color & Size
Shape & Size
Copy
5.53
6.71
1.17
6.74
1.17
1.86
VOGAN (IN-IK)
0.91
6.51
6.24
2,40
0.70
6.53
BETT (IN-22K)
15.99
9.08
1.26
7.23
2.84
2.66
MAE (IN-IK)
0.00
2.07
1.20
0.00
0.00
1.56
MAE- VQGAN (IN-1k)
0.13
2.94
3.71
0.00
0.01
3.60
VQGAN (Figures)
6.96
19.11
16.21
7.40
2.24
18.41
BEiT (Figures)
40.92
31.43
7.12
33.10
21.21
12.98
MAE (Figures)
70.23
43.99
34.72
19.30
18.99
46.02
MAE- VQGAN (Figures)
40.40
46.53
42.04
20.41
18.27
40.33
Table 3: Comparison to Fine Tuning and Classic 1-Shot Segmentation baselines. MAE-VQGAN image query and output resolution 111 1111 CyCTR and FWB resolution is 473 x433 and 5 512x512, both approach utilize Pascal 5i labeled baseclasses data.
Pretraining
Laheled Images
# Shots
Model
Split 0
Split
Split
Split
11.1
13.4
13.0
12.3
Unlabeled ImageNet
4
4
Finetune MAE
12.9
15.8
14.3
15.0
16
16
13.7
16.1
16.8
17.1
Unlabeled Figures
MAE VQGAN
32.5
33.8
32.7
27.2
Labeled Pascal 5i
1
FWB 36
51.3
64.5
56.7
52.2
(Segmentation masks)
2086
5883
1
CyCTR (5
67.2
71.1
57.6
59.0
Results. The results are presented Table for MAE-VQGAN prediction examples see Figure 5 Without training the Figures dataset, inpainting models fail to generalize to these previously unseen tasks. The performance of all models improves when they are trained on the Figures dataset. Yet. the same models struggle with combinations of tasks due to the increasing complexity. The VQGAN model utilizes sequential decoding and therefore lacks context, which leads to poor performance. The MAE model outperforms MAE-VQGE LAE-VOGAN color, and BETT performs poorly in size. These models rely on pretrained codebooks (VQGAN and dVAE) that are likely not geared towards these tasks.
30
25
15
4
5
6
7
Two support pairs
Five support pairs
Support Size
Figure 6: More examples, better results. Left: construct visual prompts with increasing number of input- output pair examples, for fixed query image (inpaintings annotated in red). Right: We observe that more examples improve the overall mIOU results on the four Pascal-Si splits.
4.4 Analysis
Comparison to finetuning and Few-Shot baselines. We include comparison to baselines that utilize K {1, 4, 16) training examples for each target class. For completeness, we also include the results of FWB (36) and CyCTR [59], classic 1-shot baselines, which we view as an upper-bound of our approach. FWB and CyCTR utilize fully labeled base classes train set (2086 to 5883 on different Pascal Si splits). Additionally, their architecture was designed for the foreground segmentation task (e.g, they operate in higher resolution). The results Table indicate that the Visual Prompting results of MAE-V LAE-VQGAN trained Figures are significantly superior to standard

---

finetuning baselines of MAEs pretrained on unlabeled ImageNet. FWB [66 and CyCTR [59] outperform Visual Prompting, mainly because they pretrain on a large tagged base classes dataset and utilize architectures that are specific to image segmentation.
Dataset effect. We evaluate the effect of pretraining on a larger and more diverse dataset. We compare training on ImageNet only, Figures only, and combination of the two, We report the mIOU results on Pascal Si for Foreground Segmentation in Figure 7 The MAE- VQGAN trained on ImageNet achieves a consistently low ~ 5 points mIOU, The model trained on the combined dataset performs best, which demonstrates that MAE-VOR VQGAN can benefit from additional amounts of unlabeled images.
34
Fiuurss
Npuan Inapelel
32
mlou
30
26
50
75
100
125
150
175
200
Training Step x 10"
Figure 7: Training MAE-YQGAN on more data improves visual prompting results.
More examples, better results. We study how increasing the number of input-output pair examples in the visual prompt affects the results. Intuitively, we expect that including more examples should reduce ambiguities and lead to better results. We use an MAE-V IQGAN pretrained on the Figures dataset, and use data from PASCAL-Si. We construct a large grid that can populate up to 8 examples and an image query. We randomly choose different numbers of examples and randomize the placements. The results in Figure c confirm that using more examples leads to better segmentation results.
Foreground Segmentation results on Pascal- 5i, when trained over the Figures dataset and on the combined Figuresand ImageNet dataset.
Prompt Engineering. We explore the effect of construct- ing different visual prompts for Foreground Segmentation and their corresponding MAE-VQGAN results (see Figure b) The model generates plausible completions when changing the prompt layout (e.g. horizontal order vs. vertical order) and when changing the mask colors, texture or using only edges (see Figure (9). The mIOU results in Table iindicat that the model performs better with a vertical layout and when the segmentation mask colors are black and white. Interestingly, by analyzing the average attention heads of a masked patch token, we observe that the attention changes following the change in the prompts layout (see Figures d-e).
Prompt Ensembling, Inspired by Prompt Ensembling in NLP (28). given the same example pair and image query. we construct multiple different visual prompts (c.g, horizontal and vertical layouts, see Figure X a-b). We then average the completion results. The results in Table 4 on the Synthetic Study tasks demonstrate that utilizing multiple prompts can lead to improved, and more stable performance.
(a) Horizontal Layout
(b) Vertical Layout
Figure 8: Prompt layout design. Two prompt orderings, and the corresponding av- erage attention maps of the selected patch (annotated with black bounding box). The highest attention values appear on similar (corresponding) areas in the query image.
Style/content extrapolation. Inspired by the classic example from Tenenbaum and Freeman 47 (on the task originally suggested by Hofstadter 24 we use MAE-VQGAN to extrapolate letter sequences printed in different fonts (see Figure T0p. We find that the model can extrapolate given style and new content (Figure IO]a but that it struggles to extrapolate new content (Figure I0p). The model also struggles to extrapolate more complex letter sequences; the performance deteriorates even if both style and content are given (Figure 10 c-d).
Limitations. The focus of this work is to present a proof of concept that shows it is possible to visually prompt simple image inpainting models trained on noisy, unlabeled data. Specifically, we demonstrate how to pre-train a network once, then prompt it to perform reasonably well on many tasks. The fact that this is possible is surprising and scientifically interesting, although this approach is not competitive with supervised task-specific models. For visual prompting to work, the inpainting models require training on the Computer Vision Figures dataset. However, our initial experiments suggest that it can benefit from training on additional natural image data (see Figure 7. Other

---

Figure 9: Task performance under different label choices. Prompting results when using different mask colors (e-g, purple/yellow vs. green/red), when drawing full mask compared to edges only, and when changing mask texture. Compared to other alternatives, purple/ycllow and black/white (sce Figure masks works best.
Table 4: Prompt Engineering, Foreground Segmentation mIOU results on Pascal-5i when using different prompt colors.
Table 5: Prompt Ensembling. We report here color-aware mIOU. In every line, the result based on ensemble of all previous prompts.
Horizontal
Vertical
Prompt Layout
Color
Shape
Horizontal
39.97
46.54
42.01
Black/White Purple/Yellow
27. 17
31.57
Vertical
41.31
54.71
46.18
23.44
28.47
Vertical w/ Rows Swap
44.14
60.42
49.42
A
A
1
2
EFGABCD EFGABCE E A CD E DCD
BCDEFG A A A B EEG A B C O D D
ABCDEFG ABCDEFG A B A B D DIO A
B
B
3
?
EFG D
B C F
(b)
Figure 10: Style and content extrapolation using MAE VQGAN, The model can extrapolate the style of a new content (a), but fails to predict a new content (b). The model strugeles to extrapolate new style and content of longer sequences (c-e).
limitations include ambiguities in the task definition, reliance on pretrained VQGAN decoder, and worse performance when the input-output example(s) are not aligned (see examples in Figure [T].
Tila LXMIFOR
Task ambiguity
Non-aligned input-output
Out of dist. decoding
Figure 11: Limitations and failure cases. Single input-output example might ambiguous can lead to unintended completions. The MAE- VQGAN model performs worse given non-aligned input-output example, and by using VQGAN vocabulary, it is limited in synthesizing oui-of-distribution pixels (like blurry images).
Discussion
Why does our proposed method, despite its simplicity, perform well on a large subset of visual tasks? At this point, we do not have good answer. Clearly, the specific training data we use plays an important role, but the amount of generalization observed still surprising. Perhaps some of these touge-to-image tasks are actually simpler than we believed. But it's also evident that contemporary large-scale inpainting models are learning quite sophisticated long-range range co-occurrences and symmetries in the data which can often enable impressive visual reasoning. We hope that our work will encourage further research better our understanding of what being learned by inpainting.
10

---

Acknowledgements: We would like to thank Assaf Shocher for insightful discussions and ideas related the Figures dataset. We thank Aaron Hertzmann, Sanjay Subramanian, Ofir Press and Ben Bogin for helpful feedback on the manuscript. This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC HOLI 819080). Prof. Darrell's group was supported in part by Dol) including DARPA's LwLL and/or SemaFor programs, as well as BAIR's s industrial alliance programs. Prof. Efros's s group was supported by in part by DoD including DARPA's MCS and/or ONR MURI, as well as funding from SAP.
References
[1] Bao, H., Dong, L., Wei, F: Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 (2021)
[2] Bar, A., Wang. X., Kantorov, V., Reed, C.J., Herzig, R., Chechik, G., Rohrbach, A., Darrell, T.,
Globerson, A.: Detreg: Unsupervised pretraining with region priors for object detection. arXiv preprint arXiw:2106.04550 (2021)
[3] Barnes, C., Shechtmamm, E., Finkelstein, A., Goldman, D.B.: Patchmatch: randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24 (2009)
[4] Bertalmio, M., Sapiro, G., Caselles, V. Ballester, C.: Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques. pp. 417-424 (2000)
[5] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Heweeeeerrrssssse A., Krueger. G.. Hemighan, T.. Child. R., Ramesh, A.. Ziegler, D.M., Wu, J.. Winter, C., Hesse, C., Chen, M., Sigler. E.. Litwin, M., Gray, S., Chess, B.. Clark,
J., Berner, C., McCandlish. S., Radford, A.. Sutskever, I., Amodei. D.: Language models are few-shot learners. CoRR ahs/2005.14165 (2020), htttss://arxi org worg/bab/ aba/ 2005/2005... 14165
[6] Caron, M., Misra, I. Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual
features by contrasting cluster assignments. NeurIPS (2020)
[7] Chang, H., Zhang. H., Jiang, I.., Liu, C., Freeman, W.T.: Maskgit: Masked generative image transformer. arXiv preprint arXiv:2202.042000 (2022)
[8] Chen, M., Radford, A., Child, R., Wu, J., Jun. H., Luan, D., Sutskever, I: Generative pretraining from
pixels. In: International Conference on Machine Learning, pp. 1691-1703, PMLR (2020)
[9] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05700 (2020)
[10] Chen, X., Fan, H. Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv (2020)
preprint arXiv:2003.0423
[11] Criminisi, A., Pérez, P., Toyama, K.; Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on image processing 13(9), 1200-1212 (2004)
[12] Devlin, J., Chang, M.W., Lee, K., Toutamova, K.; Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint arv: 10.04805 (2018)
[13] Dasovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929 (2020)
[14] Efros, A.A., Leung, T.K.: Texture synthesis hy non-parametric sampling. In: IEFE International Conference on Computer Vision. pp. 1033-1038. Corfu, Greece (September 1999)
[15] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. -1873-1883 (2021)
[16] Gidaris, S., Bursuc, A., Komodakis, N., Pérez, P., Cord, M.: Learning representations by predicting bags of visual words. In: CVPR (2020)
[17] Goodfellow, I., Abadit-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural information processing systems 27 (2014)
[18] Goyal, P., Mahajan, D., Gupta, A., Misra, L: Scaling and benchmarking self-supervised visual representa- tion learning. In: ICCV (2019)
[19] Hays, J., Efros, A.A.: Scene completion using millions of photographs. ACM Transactions Graphics (SIGGRAPH 2007) 26(3) (2007)
[20] He. K. Chen, X., Xie, S., Li, Y. Dollár, P., Girshick, R.B.: Masked autoencoders are scalable vision learners. CoRR abs/2111.06377 (2021), https::/arxxvv. org.aas..rrr/b.a...rrb./o wtt.o//////////.....eee 22111.0677
11

---

[21] He. K., Fan, H., Wu, Y.. Xie, S., Cirshick, R.: Momentum contrast for unsupervised visual representation learning. In: CVPR (2020)
[22]
Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of
the 28th annual conference on Computer graphics and interactive techniques. pp. 327-340 (2001)
[23] Hill, F., Santoro, A., Barrett, D.G., Morcos, A.S., Lillicrap, T.: I Learning to make analogies by contrasting abstract relational structure. arXiv preprint arXiv: 190..00000000220000 (2019)
[24] Hofstadter, D.R.: Fluid concepts and creative analogies: Computer models of the fundamental mechanisms of thought. Basic books (1995)
[25] Jiang. Z., Xu, F.F. Araki, J., Neubig, G.: How we know what language models know? Transactions of the Association for Computational Linguistics 8, 423 438 (2020)
[26] Johnson, J. Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In:
European conference computer vision. pp. 694-711. 1. Springer (2016)
[27] Kang. B., Liu. Z., Wang. X.. Yu, F., Feng, J., Darrell, T.: Few-shot object detection via feature reweighting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8420-8422 8449 (2019)
[28] Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. CoRR abs/2104.08691
org/abs/ ht////bs/21000 08691
[29] Li, X.I.., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)
[30] Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: using partial convolutions. In: Proceedings of the European conference on computer vision (ECCV). pp. 85-100 (2018)
Image inpainting for irregular holes
[31] Liu, G., Reda, F.A., Shih, K.J., Wang, T.C.,, Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV) (September 2018)
[32] Liu, Y., Zhang, X., Zhang, S., Hc, X.: Part-aware prototype network for few-shot semantic segmentation. In: European Conference Computer Vision. pp. 142-158. Springer (2020)
[33] Lu, Y., Bartolo, M.. Moore, A., Riedel, S., Stenetorp, P:: Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint 11X0:22044008866 (2021)
[34] Memisevic, R., Hinton, G.: Unsupervised learning of image transformations. In: 2007 IEEE Conference
on Computer Vision and Pattern Recognition. pp. 1-8. IEFE (2007)
[35] Misra, I., Maaten, L.v.d. Self-supervised learning of pretext-invariant representations. In: CVPR (2020)
[36] Nguyen, K., Todorovic, S.: Feature weighting and boosting for few-shot segmentation. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 622 661 (2019)
[37] Van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv e-prints pp. arXiv-1807 (2018)
[38] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2536-2544 (2016)
[39] Radford, A., Wu, J.. Child, R., Luan, D., Amodei, D., Sutskever, et et al.: Language models are unsuper- vised multitask learners. OpenAl blog 1(8), 9 (2019)
[40] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, L.: Zero-shot text-to-image generation. In: International Conference on Machine Learning, pp. 8821-8831, PMLR (2021)
[41] Reed, S.E., Zhang, Y., Zhang, Y., Lee, H.: Deep visual analogy-making Advances in neural information processing systems 28 (2015)
[42] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma. S., Huang. Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (ICVV 115(3), 21 211-252 (2015). https:///oi.org//////// wrtp./o.///////////.. 1263-015-088644
[43] Sadeghi, F. Zitnick, C.I.,, Farhadi, A.: Visalogy: Answering visual analogy questions. Advances in Neural Information Processing Systems 28 (2015)
[44] Sarzynska-Wann J., Wawer, A., Pawlak, A., Szymanowska, J., Stefaniak, I., Jarkiewicz, M., Okruszek,
L.: Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research 304, 114135 (2021)
[45] Shaban, A.. Bansal, S., Liu, Z., Essa, I,, Boots, B.: shot learning for semantic segmentation. CoRR abs/1709.03410 (2017), http: //tp///rvvvv..////////eee/eaaalr warg//bb///ie http://1990044440 03410
12

---

[46] Taylor, G.W., Fergus, R., LeCun, Y., Bregler, C.: Convolutional learning of spatio-temporal features. In: European conference computer vision. pp. 140-153. Springer (2010)
[47] Tenenbaum, J.B., Freeman, W.T.: Separating style and content with bilinear models. Neural computation
12(6). 1247- 1283 (2000)
[48] Tian. Z.. Zhao, Shu, M.. Yang, Z., R., Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence (2020)
[49] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. arXiv:1711. 10925 (2017)
[50] Upchurch, P., Snavely, N., Bala, K.: From a to z: supervised transfer style and content using deep neural network generators. arXiv preprint arXiv: 1603.02003 (2016)
[51] Van Den Oord, Vinyals,
Neural discrete representation learning. Advances neural
information processing systems 30 (2017)
[52] Vaswani, A., Shazcer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.,, Polosukhin, I.:
Attention you need. Advances in neural information processing systems (2017)
[53] Wang, X., Huang. T.E., Darrell, T., Gonzalez, J.F., Yu, F:: Frustratingly simple few-shot object detection.
arxiv preprint arXiv:2003.06955 (2020)
[54] Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., Hu, H.: Simmim: A simple framework for masked image modeling. Xiviv preprint arXiv:211 1.09886 (2021)
[55] Yang, B., Liu, C., Li, B., Jiao, Ye, Q.: Prototype mixture models for few-shot semantic segmentation. In: European Conference on Computer Vision. pp- 763-778. Springer (2020)
[56] Yang, C.. Lu, X., Lin, 7., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using
multi-scale neural patch synthesis. In: Proceedings of the IEEE conference computer vision and pattern
recognition. pp. 6721-6729 (2017)
[57] Yu. Li, X., Koh, J.Y. Zhang, H. Pang. R., Qin, J., Ku, A., Xu, Y., Baldridge, J., Wu, Y.: Vector-quantized image modeling with improved vggan, arXiv preprint atXiv:211000666 (2021)
[58] Yu. Y., Zhan, F. Wu, R., Pan, J., Cui, K., Lu, S., Ma, F,, Xie, X., Miao, C.: Diverse image inpainting with
bidirectional and autoregressive transformers. In: Proceedings of the 29th ACM International Conference Multimedia. pp. 69-78 (2021)
[59] Zhang, G., Kang, G., Yang, Y., Wei, Y.: Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems 34 (2021)
[60] Zhang, R., Isola, Efros, A.A., Shechtman, E., Wang. O.: The unrcasonable effectiveness deep features as perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586-595 (2018)
13


Reviews:
[{"review_id": "p54Qo0FPIyf", "reviewer": "Reviewer_WRn2", "summary": "The paper proposed a dataset constructed of figures from computer vision and machine learning arXiv papers filtered for grid-style figures. This allows training an image in-painting model on a large variety of input-output examples (contained in one image). The model can then be used to solve several popular tasks by “prompting” in the sense that an image is constructed as a grid of input-output examples with the last output being masked for in-painting. The model prediction can then be interpreted as a task prediction. \nThe paper contains a broad evaluation of different architectures, tasks, and prompts and shows that in many arrangements the final model produces good results when trained on this dataset vs. ImageNet.\n", "questions": "The questions here directly reflect the weaknesses above.\n1. What is the definition of few-shot learning in this paper and how does it relate to the existing approaches of few-shot learning?\n2. How can we make sure that the evaluation of the model uses unseen categories/tasks/images if the dataset is sourced from results of other papers?\n3. How was the dataset collected? What does it contain? Were individual licenses taken into account?\n4. Does the model pick up a bias in artificially predicting imprecise results to emulate predictions?", "limitations": "The paper contains a section on technical limitations of the work but does not discuss the societal impact of the work nor the dataset. A datasheet is strongly encouraged as well as a more clear description on how the dataset was collected and filtered. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper shows a straight-forward way to train an inpainting model to be used for visual prompting at inference time. This idea is quite interesting and the trained model shows reasonable performance on several tasks.\n\nThe paper evaluates several architectures, training datasets, prompting schemes and tasks, and analyzes all major design choices to show that the final model performs best.\n\nThe proposed dataset of figures from arxiv papers and could be of broader interest to the research community.\n\n\nWeaknesses\n\nFew-shot learning: The paper claims that models trained like that are few-shot learners. However, this statement is based on a very stretched definition of few-shot learning. The most common example of few-shot learning in computer vision is classification, where a model is trained on a dataset of training classes with images. During test time, the model is shown a few annotated examples of a previously unseen class and has to produce a classifier that can recognize this class. The important difference here is that in this setup (and others such as few-shot detection, semantic segmentation, etc.) is that while the task remains the same, the model is presented with unseen categories at test time. In this paper the setup is quite different, as the evaluation does not test unseen categories. Without a proper analysis of the dataset (which is missing) it is unclear which categories, tasks and even images have been seen during training. As most of the tasks that are evaluated in the paper are highly popular computer vision tasks with hundreds of papers on arxiv (segmentation, colorization, detection, inpainting, edge detection) there is a high chance that the model simply recognizes the task from the prompt instead of learning a new category or a new task from the given examples.\n\nDataset: There is very little information about the collected dataset in the paper and appendix which makes it not only difficult to assess the findings of the paper but also leaves many questions on the dataset itself. I strongly suggest creating a “datasheet for datasets” for this dataset. For example: it is not clear if the license of the arxiv papers has been taken into account before extracting the figures. It is unclear if copyright and attributions are retained in the dataset. By nature, the dataset will consist of many validation/testing images from popular datasets, thus training a model on this data will likely skew its performance when evaluated on the val/test sets for downstream task performance.\n\nLearning Signal: This point is related to the content of the dataset. Figures in papers often compare the results of the method with the ground truth and other methods. With this prompting approach it could be possible that a model learns to predict imprecise results from the fact that papers most often show predictions. It could be interesting to investigate how much this affects the method. Potential ideas could be adding labels (e.g. “GT”, “pred”, “ours” to the columns).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)", "Legal Compliance (e.g., GDPR, copyright, terms of use)"]}, "extra_text": {}}, {"review_id": "lS_JXTtMECsQ", "reviewer": "Reviewer_xQdx", "summary": "This paper proposes a uniform framework for various image tasks, such as colorization, edge detection, inpainting, segmentation, and style transfer. This paper treats different image tasks as an inpainting task and achieves good performance. The authors also collect a dataset for training models.", "questions": "How did you combine the MAE and VQGAN? During training, did you train VQGAN first? And then use MAE to reconstruct the visual tokens? During inference, do you encode the whole combined image using VQGAN first? And then use MAE to fill the missing part (visual tokens instead of pixels)? And then use VQGAN decoder to reconstruct the image? \n\nWhy do you use MAE to fill the missing parts? Why not just use the transformer from VQGAN to generate the missing parts directly? Is the MAE part necessary?\n\nHow do you assign probabilities to visual tokens in L111? \n\nDoes the prediction of z^{\\hat}_k related to z^{\\hat}_{k-1} in the eqn under L113? Does each z^{\\hat}_k are predicted only based on x and m? \n\nIn the eqn right above L119, is m a mask of the part needs to be filled?\n\nIf the authors can address all the above questions well, I think this is a good paper to be accepted. \n\n", "limitations": "Some model details are missing. For example, the model details of MAE and VQGAN as mentioned above. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) The authors propose a smart way to solve various image tasks by treating them as an inpainting task. This paper is very impressive. The idea is novel and the paper is well-written. The results are convincing. \n\n(-) Some method details and implementation details can be added to improve this paper.\n\n(-) This paper did not compare with the standard methods for image segmentation, colorization, object detection, etc. For example, Mask-RCNN [1], Mask2Former[2], [3] and so on. The authors should add comparison with these standard methods and the state-of-the-art methods in each domain.\n\n[1] Mask R-CNN\n[2 ] Masked-attention Mask Transformer for Universal Image Segmentation\n[3] Real-Time User-Guided Image Colorization with Learned Deep Priors\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PFLRw_rmwCZ", "reviewer": "Reviewer_HDgS", "summary": "The paper aims to generalize the idea of natural language prompting to the vision domain. They propose visual prompting by image inpainting. Given task examples (image, target) and a image query, they construct a new “grid-like image” (i.e., visual prompt) and the inpainting model predicts the masked region such that it is consistent with the example(s). Given that it was trained on the right data, the inpainting model can perform prompting in various vision tasks. One key assumption is that the tasks can be defined as an image-to-image translation problem. To train the in-painting model, they collect a dataset of 88k figures, which is critical for performance.", "questions": "* How does the method compare to baseline methods (state-of-the-art, fine-tuning, zero shot)? In fact, what is the practical utility of this prompting method? The benefit of prompting is the ability to generalize to a new task *without additional training*. However, the proposed method requires having to train a whole new model which essentially contradicts the value of prompting. If the prompting method only works on the proposed pre-trained model, how useful/general is this pre-trained model? Does the proposed pre-training + prompting method outperform training from scratch using the same dataset/model with a task-specific state-of-the-art objective? If not, why do we have to use this prompting method?\n\n* I’m confused about how the datasets (ImageNet, Figures) are processed during pre-training. Are they concatenated as a grid-like image and processed as input or is only a single image used as input? From what I understand, during pre-training, a single image is used as input and a random region is masked, and the model is trained to reconstruct the image. I believe this is inefficient. For prompting, closely matching the pre-trained and downstream datasets is key. Therefore, rather than collecting a whole new dataset (Figures) and do random region masking, can’t you reformulate existing datasets (e.g. ImageNet) into a grid-like structure and pre-train the model? Have you tried this? How does this performance compare to having the models pre-trained on the Figures Dataset with random masking? This would further reduce the domain gap between pre-training and downstream tasks. ", "limitations": "I think the key limitations of this work are domain gap between pre-trained and downstream task,  scalability (the pre-trained model specifically requires figure-like training images), and possibly performance (does this method outperform training from scratch using state-of-the-art?). While I really like the pure vision approach for prompting, these limitations make me question the practical utility of this work. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* It is very interesting to see that in-context learning in NLP works for images, too. While the set-up is extremely simple, the paper shows that pre-trained task of image in-painting is flexible enough to cover a wide variety of vision tasks.\n* It is one of the first papers that shows that in-context learning works for vision. While other works incorporate vision into a language modeling task or focus on using language prompts, I like that this paper takes a “purely vision” approach. One value of this paper is showing that prompting can be effective for vision *without* using language pre-training tasks or language prompts. \n* Overall, the paper is well-written and easy to follow.\n\nWeaknesses\n* Having to collect a whole new dataset (Figures) specific to this pre-trained task lacks scalability, when having a sufficiently large training set is crucial for creating a high-quality pre-trained model. \n* In fact, how the authors perform pre-training seems inefficient. For prompting, closely matching the pre-trained and downstream tasks is key. While their prompt for the downstream tasks is “grid-like images” with support examples and a query image, the pre-training task is in-painting with random region masking. This creates an unnecessary domain gap between pre-training and downstream tasks. Can’t we just reformulate existing datasets (e.g. ImageNet) into a grid-like structure and use them for pre-training? \n* The paper lacks baseline comparisons. For Table 1, how does the method compare to state-of-the-art, fine-tuning, and zero shot? It is very difficult to assess the value of the proposed method without any comparisons. Essentially, prompting is a method that allows *no additional training*. This paper requires *pre-training a whole new model* specific to the proposed prompting method which contradicts the value of prompting. Have you tried using the prompt on any off-the-shelf pre-trained in-painting model (i.e., zero shot)? Or such a pre-trained model doesn't exist for the proposed prompting method?\n* For colorization, you should measure LPIPS [1] instead of MSE. MSE cannot accurately measure the perceptual similarity between colored output and the target.\n\n[1] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, Zhang et al., CVPR 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OKdVWUlnpkD", "reviewer": "Reviewer_v9Mo", "summary": "This paper proposes a novel approach to use a pre-trained visual model to various downstream tasks via “prompting”. In other words,  it can be viewed as a one-shot or few-shot learner but without the need for fine-tuning. Similar to NLP systems, they propose to use image inpainting as the pre-training task. To prompt for a downstream task, the method concatenates the exemplar pair images and the input image into a larger “visual prompt image”. Notably, they demonstrate that the dataset that the visual model is pre-trained on is important. Specifically, they collected images of paper figures from arXiv papers. Empirically, they demonstrate their approach to various downstream tasks, including foreground segmentation, single object detection, colorization. ", "questions": "See #2, #3, #4 from above.", "limitations": "The paper contains a limitation section and adequately discussed the shortcomings of the approach. Specifically, the inherit ambiguity when prompting from a single image for a task. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe proposed approach is novel and interesting. The authors proposed a simple (in a good way) approach to enable prompting on visual models with many analogues to NLP. Personally, I find the insight to training on arXiv paper figures intuitive and creative. Overall, the writing is organized and clear. Finally, the authors also promise to release the data and code, which would benefit the community.\n# Weaknesses\n#### 1. I believe the paper would benefit from a comparison with a one-shot baseline. While the proposed approach does not require fine-tuning, it would be interesting to see how it competes.\n#### 2. The prompt engineering details are unclear. I was only able to get a vague idea on how to construct the visual prompt. Is it possible to document them more precisely? Additionally, how are different image sizes being handled? Are the images being resized? \n#### 3. Missing training details? Specifically, I am wondering if the VQGAN is pre-trained? Or only trained on the 88,635 images from the Computer Vision Figures dataset.\n#### 4. The authors have stated that they repeated the experiment with four random seed (line. 194). It would be great to report the standard deviation on the quantitative results. Also, I wonder how sensitive the approach is to the prompted image. For example, would the approach still work if the cat, in Fig.3, is prompted by an image of a white cat that's outdoor?\n\n# Misc.\n- Line 173: \"224x224\" --> \"224 \\times 224\"\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept