Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel approach to use a pre-trained visual model to various downstream tasks via “prompting”. In other words,  it can be viewed as a one-shot or few-shot learner but without the need for fine-tuning. Similar to NLP systems, they propose to use image inpainting as the pre-training task. To prompt for a downstream task, the method concatenates the exemplar pair images and the input image into a larger “visual prompt image”. Notably, they demonstrate that the dataset that the visual model is pre-trained on is important. Specifically, they collected images of paper figures from arXiv papers. Empirically, they demonstrate their approach to various downstream tasks, including foreground segmentation, single object detection, colorization. 
strengths_and_weaknesses: # Strengths
The proposed approach is novel and interesting. The authors proposed a simple (in a good way) approach to enable prompting on visual models with many analogues to NLP. Personally, I find the insight to training on arXiv paper figures intuitive and creative. Overall, the writing is organized and clear. Finally, the authors also promise to release the data and code, which would benefit the community.
# Weaknesses
#### 1. I believe the paper would benefit from a comparison with a one-shot baseline. While the proposed approach does not require fine-tuning, it would be interesting to see how it competes.
#### 2. The prompt engineering details are unclear. I was only able to get a vague idea on how to construct the visual prompt. Is it possible to document them more precisely? Additionally, how are different image sizes being handled? Are the images being resized? 
#### 3. Missing training details? Specifically, I am wondering if the VQGAN is pre-trained? Or only trained on the 88,635 images from the Computer Vision Figures dataset.
#### 4. The authors have stated that they repeated the experiment with four random seed (line. 194). It would be great to report the standard deviation on the quantitative results. Also, I wonder how sensitive the approach is to the prompted image. For example, would the approach still work if the cat, in Fig.3, is prompted by an image of a white cat that's outdoor?

# Misc.
- Line 173: "224x224" --> "224 \times 224"

questions: See #2, #3, #4 from above.
limitations: The paper contains a limitation section and adequately discussed the shortcomings of the approach. Specifically, the inherit ambiguity when prompting from a single image for a task.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper aims to generalize the idea of natural language prompting to the vision domain. They propose visual prompting by image inpainting. Given task examples (image, target) and a image query, they construct a new “grid-like image” (i.e., visual prompt) and the inpainting model predicts the masked region such that it is consistent with the example(s). Given that it was trained on the right data, the inpainting model can perform prompting in various vision tasks. One key assumption is that the tasks can be defined as an image-to-image translation problem. To train the in-painting model, they collect a dataset of 88k figures, which is critical for performance.
strengths_and_weaknesses: Strengths
* It is very interesting to see that in-context learning in NLP works for images, too. While the set-up is extremely simple, the paper shows that pre-trained task of image in-painting is flexible enough to cover a wide variety of vision tasks.
* It is one of the first papers that shows that in-context learning works for vision. While other works incorporate vision into a language modeling task or focus on using language prompts, I like that this paper takes a “purely vision” approach. One value of this paper is showing that prompting can be effective for vision *without* using language pre-training tasks or language prompts. 
* Overall, the paper is well-written and easy to follow.

Weaknesses
* Having to collect a whole new dataset (Figures) specific to this pre-trained task lacks scalability, when having a sufficiently large training set is crucial for creating a high-quality pre-trained model. 
* In fact, how the authors perform pre-training seems inefficient. For prompting, closely matching the pre-trained and downstream tasks is key. While their prompt for the downstream tasks is “grid-like images” with support examples and a query image, the pre-training task is in-painting with random region masking. This creates an unnecessary domain gap between pre-training and downstream tasks. Can’t we just reformulate existing datasets (e.g. ImageNet) into a grid-like structure and use them for pre-training? 
* The paper lacks baseline comparisons. For Table 1, how does the method compare to state-of-the-art, fine-tuning, and zero shot? It is very difficult to assess the value of the proposed method without any comparisons. Essentially, prompting is a method that allows *no additional training*. This paper requires *pre-training a whole new model* specific to the proposed prompting method which contradicts the value of prompting. Have you tried using the prompt on any off-the-shelf pre-trained in-painting model (i.e., zero shot)? Or such a pre-trained model doesn't exist for the proposed prompting method?
* For colorization, you should measure LPIPS [1] instead of MSE. MSE cannot accurately measure the perceptual similarity between colored output and the target.

[1] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, Zhang et al., CVPR 2018.
questions: * How does the method compare to baseline methods (state-of-the-art, fine-tuning, zero shot)? In fact, what is the practical utility of this prompting method? The benefit of prompting is the ability to generalize to a new task *without additional training*. However, the proposed method requires having to train a whole new model which essentially contradicts the value of prompting. If the prompting method only works on the proposed pre-trained model, how useful/general is this pre-trained model? Does the proposed pre-training + prompting method outperform training from scratch using the same dataset/model with a task-specific state-of-the-art objective? If not, why do we have to use this prompting method?

* I’m confused about how the datasets (ImageNet, Figures) are processed during pre-training. Are they concatenated as a grid-like image and processed as input or is only a single image used as input? From what I understand, during pre-training, a single image is used as input and a random region is masked, and the model is trained to reconstruct the image. I believe this is inefficient. For prompting, closely matching the pre-trained and downstream datasets is key. Therefore, rather than collecting a whole new dataset (Figures) and do random region masking, can’t you reformulate existing datasets (e.g. ImageNet) into a grid-like structure and pre-train the model? Have you tried this? How does this performance compare to having the models pre-trained on the Figures Dataset with random masking? This would further reduce the domain gap between pre-training and downstream tasks. 
limitations: I think the key limitations of this work are domain gap between pre-trained and downstream task,  scalability (the pre-trained model specifically requires figure-like training images), and possibly performance (does this method outperform training from scratch using state-of-the-art?). While I really like the pure vision approach for prompting, these limitations make me question the practical utility of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed a dataset constructed of figures from computer vision and machine learning arXiv papers filtered for grid-style figures. This allows training an image in-painting model on a large variety of input-output examples (contained in one image). The model can then be used to solve several popular tasks by “prompting” in the sense that an image is constructed as a grid of input-output examples with the last output being masked for in-painting. The model prediction can then be interpreted as a task prediction. 
The paper contains a broad evaluation of different architectures, tasks, and prompts and shows that in many arrangements the final model produces good results when trained on this dataset vs. ImageNet.

strengths_and_weaknesses: Strengths

The paper shows a straight-forward way to train an inpainting model to be used for visual prompting at inference time. This idea is quite interesting and the trained model shows reasonable performance on several tasks.

The paper evaluates several architectures, training datasets, prompting schemes and tasks, and analyzes all major design choices to show that the final model performs best.

The proposed dataset of figures from arxiv papers and could be of broader interest to the research community.


Weaknesses

Few-shot learning: The paper claims that models trained like that are few-shot learners. However, this statement is based on a very stretched definition of few-shot learning. The most common example of few-shot learning in computer vision is classification, where a model is trained on a dataset of training classes with images. During test time, the model is shown a few annotated examples of a previously unseen class and has to produce a classifier that can recognize this class. The important difference here is that in this setup (and others such as few-shot detection, semantic segmentation, etc.) is that while the task remains the same, the model is presented with unseen categories at test time. In this paper the setup is quite different, as the evaluation does not test unseen categories. Without a proper analysis of the dataset (which is missing) it is unclear which categories, tasks and even images have been seen during training. As most of the tasks that are evaluated in the paper are highly popular computer vision tasks with hundreds of papers on arxiv (segmentation, colorization, detection, inpainting, edge detection) there is a high chance that the model simply recognizes the task from the prompt instead of learning a new category or a new task from the given examples.

Dataset: There is very little information about the collected dataset in the paper and appendix which makes it not only difficult to assess the findings of the paper but also leaves many questions on the dataset itself. I strongly suggest creating a “datasheet for datasets” for this dataset. For example: it is not clear if the license of the arxiv papers has been taken into account before extracting the figures. It is unclear if copyright and attributions are retained in the dataset. By nature, the dataset will consist of many validation/testing images from popular datasets, thus training a model on this data will likely skew its performance when evaluated on the val/test sets for downstream task performance.

Learning Signal: This point is related to the content of the dataset. Figures in papers often compare the results of the method with the ground truth and other methods. With this prompting approach it could be possible that a model learns to predict imprecise results from the fact that papers most often show predictions. It could be interesting to investigate how much this affects the method. Potential ideas could be adding labels (e.g. “GT”, “pred”, “ours” to the columns).

questions: The questions here directly reflect the weaknesses above.
1. What is the definition of few-shot learning in this paper and how does it relate to the existing approaches of few-shot learning?
2. How can we make sure that the evaluation of the model uses unseen categories/tasks/images if the dataset is sourced from results of other papers?
3. How was the dataset collected? What does it contain? Were individual licenses taken into account?
4. Does the model pick up a bias in artificially predicting imprecise results to emulate predictions?
limitations: The paper contains a section on technical limitations of the work but does not discuss the societal impact of the work nor the dataset. A datasheet is strongly encouraged as well as a more clear description on how the dataset was collected and filtered.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a uniform framework for various image tasks, such as colorization, edge detection, inpainting, segmentation, and style transfer. This paper treats different image tasks as an inpainting task and achieves good performance. The authors also collect a dataset for training models.
strengths_and_weaknesses: (+) The authors propose a smart way to solve various image tasks by treating them as an inpainting task. This paper is very impressive. The idea is novel and the paper is well-written. The results are convincing. 

(-) Some method details and implementation details can be added to improve this paper.

(-) This paper did not compare with the standard methods for image segmentation, colorization, object detection, etc. For example, Mask-RCNN [1], Mask2Former[2], [3] and so on. The authors should add comparison with these standard methods and the state-of-the-art methods in each domain.

[1] Mask R-CNN
[2 ] Masked-attention Mask Transformer for Universal Image Segmentation
[3] Real-Time User-Guided Image Colorization with Learned Deep Priors

questions: How did you combine the MAE and VQGAN? During training, did you train VQGAN first? And then use MAE to reconstruct the visual tokens? During inference, do you encode the whole combined image using VQGAN first? And then use MAE to fill the missing part (visual tokens instead of pixels)? And then use VQGAN decoder to reconstruct the image? 

Why do you use MAE to fill the missing parts? Why not just use the transformer from VQGAN to generate the missing parts directly? Is the MAE part necessary?

How do you assign probabilities to visual tokens in L111? 

Does the prediction of z^{\hat}_k related to z^{\hat}_{k-1} in the eqn under L113? Does each z^{\hat}_k are predicted only based on x and m? 

In the eqn right above L119, is m a mask of the part needs to be filled?

If the authors can address all the above questions well, I think this is a good paper to be accepted. 


limitations: Some model details are missing. For example, the model details of MAE and VQGAN as mentioned above.
review_title: null

Meta-review: metareview: The paper discusses a way to use pre-trained models for downstream tasks. Reviewers generally appreciated the paper but had questions regarding baselines, details, dataset, etc. The rebuttal addressed most of these concerns prompting the reviewers to raise their recommendation. However some questions remained (e.g., https://openreview.net/forum?id=o4uFFg9_TpV&noteId=wPCrVV96hE). AC concurs with the unanimous reviewer recommendation.
confidence: Certain
Decision: Accept