Title: What Makes Good Examples for Visual In-Context Learning?
Venue: NeurIPS 2023
Accepted: yes
Keywords: computer vision

visual in-context learning

prompt learning
Abstract: Large vision models with billions of parameters and trained on broad data have great potential in numerous downstream applications. However, these models are typically difficult to adapt due to their large parameter size and sometimes lack of accesss to their weights---entities able to develop large vision models often provide APIs only. In this paper, we study how to better utilize large vision models through the lens of in-context learning, a concept that has been well-known in natural language processing but has only been studied very recently in computer vision. In-context learning refers to the ability to perform inference on tasks never seen during training by simply conditioning on in-context examples (i.e., input-output pairs) without updating any internal model parameters. To demystify in-context learning in computer vision, we conduct an extensive research and identify a critical problem: downstream performance is highly sensitivie to the choice of visual in-context examples. To address this problem, we propose a prompt retrieval framework specifically for large vision models, allowing the selection of in-context examples to be fully automated. Concretely, we provide two implementations: (i) an unsupervised prompt retrieval method based on nearest example search using an off-the-shelf model, and (ii) a supervised prompt retrieval method, which trains a neural network to choose examples that directly maximize in-context learning performance. Both methods do not require access to the internal weights of large vision models. Our results demonstrate that our methods can bring non-trivial improvements to visual in-context learning in comparison to the commonly-used random selection. Code and models will be released.

What Makes Good Examples for Visual In-Context Learning?

Abstract

Large vision models with billions of parameters and trained on broad data have great potential in numerous downstream applications. However, these models are typically difficult to adapt due to their large parameter size and sometimes lack of accesss to their weights-entities able to develop large vision models often provide APIs only. In this paper, we study how to better utilize large vision models through the lens of in-context learning, a concept that has been well-known in natural language processing but has only been studied very recently in computer vision. In-context learning refers to the ability to perform inference on tasks never seen during training by simply conditioning on in-context examples (i.e., input-output pairs) without updating any internal model parameters. To demystify in-context learning in computer vision, we conduct an extensive research and identify a critical problem: downstream performance is highly sensitivie to the choice of visual in-context examples. To address this problem, we propose a prompt retrieval framework specifically for large vision models, allowing the selection of in-context examples to be fully automated. Concretely, we provide two implementations: (i) an unsupervised prompt retrieval method based on nearest example search using an off-the-shelf model, and (ii) a supervised prompt retrieval method, which trains a neural network to choose examples that directly maximize in-context learning performance. Both methods do not require access to the internal weights of large vision models. Our results demonstrate that our methods can bring non-trivial improvements to visual in-context learning in comparison to the commonly-used random selection.

1 Introduction

In recent years, large-scale models have emerged in computer vision: they have enormous parameter sizes and are pre-trained on broad data to gain wide-ranging knowledge. These models have demonstrated remarkable generalization performance and have great potential for numerous downstream applications [4]. However, due to the large model size and the potentially proprietary data used for training, entities able to develop large-scale models typically only provide users with APIs, known as Model-as-a-Service (Maas). Representative examples include the prominent text-to-image generation models, DALL · E [17] and Imagen [20], and OpenAI's powerful language models like GPT3/ChatGPT [16]. As a result, users are unable to apply full fine-tuning or some parameter-efficient tuning techniques, such as prompt learning [11, 10, 26, 25, 23, 15], for model adaptation, largely limiting downstream performance. To circumvent the problem, 'black-box' adaptation methods need to be developed.

0

20

40

60

80

100

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Foreground segmentation (mIoU)

Average

Standard deviation

Best prompt

Worst prompt

…

…

Example

Annotation

Query

Output

Database (train)

Query (test)

Large-scale

vision model

(b) Prompt retrieval for visual in-context learning

Score

function

Retrieved

image

(a) Visual in-context learning is sensitive to prompt selection

Query image index

fe(·

In-context learning , which is a 'hidden' capability originally found in large autoregressive language models [16], has recently been investigated for large vision models [3], and more importantly, has the potential to be adopted as the mainstream approach in MaaS applications in the future. Without the need to update any parameter when deploying in previously unseen tasks, in-context learning simply prepends some domain-specific input-output pairs, called in-context examples or prompt, 2 to a test example, which together guide the model to produce an ideal result. For instance, in natural language processing one could prepend a French-English sentence pair to a French sentence, and the model would produce an English translation of the French sentence. In computer vision, Bar et al. . [3] pre-trained a neural network to fill missing patches in grid-like images, which allows the model to perform in-context learning for unseen tasks like image segmentation (see the grid images in Fig. 1(a) bottom).

In this work, we focus on visual in-context learning , a relatively new concept with little existing research regarding how to better apply it in practice. We for the first time conduct a comprehensive investigation on the impact of in-context examples for large vision models, and identify a critical issue: downstream performance is highly sensitive to the choice of in-context examples. This is evidenced by the large variances observed for a variety of test examples shown in Fig. 1(a) top. By visualizing the results in Fig. 1(a) bottom, it seems to suggest that the closer the in-context example to the query, the better the result. For example, the best prompt image is closer to the query as they are similar in object pose and background; on the other hand, the worst prompt image has a drastically different style than the query image, which might explain why the predicted mask focuses on the wrong region, i.e., the white pillar instead of the cat.

Clearly, designing a proper prompt containing the optimal in-context example(s) by hand would be extremely difficult. To overcome the problem, we propose a prompt retrieval framework where the core component is a score function, which aims to give each source instance a score to indicate the level of suitability for being included in the prompt. Once the scoring process is done, we can simply pick one or multiple examples with the highest score(s) to construct a prompt. An overview of our framework is depicted in Fig. 1(b).

We provide two implementations for the prompt retrieval framework, both interpreting the score as the cosine distance measuring similarity between a query and a source example. The first is an unsupervised method based on nearest example search using an off-the-shelf model. The second

Feature

space

Learnable

feature

extractor

…

…

Query

🔥

🧊

Large-scale

vision model

Query (train)

🔥

Randomly sampled

prompt candidates

(train)

Inpainted

result

Prompt

Example

Annotation

Query

Mask

IoU = (GT-Inpainted)

GTresult

of query

Positives:

Examples with

high IoU

Negatives:

Examples with

low IoU

is a supervised method, which learns a neural network to choose examples that directly maximize in-context learning performance. Since there is no ground-truth score to be used as the supervisory signal, we resort to a contrastive learning paradigm: source examples that result in better (or worse) in-context learning performance should get closer (or farther) to the query in feature space.

Our contributions and the main findings are summarized as follows. (1) We present the first comprehensive study concerning how to select good examples for the emerging visual in-context learning, and reveal a critical issue that the choice of in-context examples has a huge impact on performance. (2) From the technical perspective, we present a prompt retrieval framework that can automate the prompt selection process, and provide two simple implementations: an unsupervised method and a supervised method. (3) By conducting extensive experiments on three visual in-context learning tasks (which have not been seen during pre-training), namely foreground segmentation, single object detection and image colorization, we share valuable insights with the community on how to find good visual in-context examples, e.g., the supervised method performs the best and often finds examples that are both semantically close and spatially similar to a query. (4) The code and models are available at https://github.com/ZhangYuanhan-AI/visual_ prompt_retrieval .

2 Methods

2.1 Visual In-Context Learning

In-context learning is a new paradigm that originally emerged from large autoregressive language models pre-trained on broad data, such as GPT-3 [5]. Unlike traditional learning methods, in-context learning does not require any parameter update and instead conditions prediction on some in-context examples in the form of input-output pairs. For example, in natural language processing one might give a French-English sentence pair and a test French sentence as input to the model, which then produces the English version of the sentence. In computer vision, such a paradigm has only been studied very recently. For example, Bar et al. [3] trained a neural network to fill missing patches in grid-like images, which in turn allows the model to perform in-context learning on unseen tasks.

Formally, given a dataset D = { ( x n , y n ) } N n =1 containing N image-label pairs (e.g., an image and its segmentation mask), a query example x q , and a model g τ , in-context learning can be formulated as:

where P is called a prompt, which consists of K input-output pairs, P = { x c 1 , y c 1 , ..., x c K , y c K } ⊂ D . In particular, the prompt P provides some context for guiding the model to produce the ideal y q for x q without updating the large model's parameters τ .

Problem. The most common approach for designing the prompt P in the vision domain is (withinclass) random selection proposed by [3]: one or multiple image-label pairs (with the same label as the test example) are randomly chosen from the training dataset. As illustrated in Fig. 1(a), the

performance is highly sensitive to the selection of in-context examples-the gap between the best and worst prompt could reach over 70% mIoU. Below we propose two automatic prompt selection methods to tackle this problem.

2.2 Prompt Retrieval

Our goal is to automatically select the most suitable example(s) from the training dataset for a query x q . To this end, we propose a prompt retrieval framework in the following form,

where f θ is a function parameterized by θ , aiming to produce a score for a pair of x n and x q . When K = 1 , we choose the optimal example pair as the prompt, P = { x ∗ , y ∗ } . When K > 1 , we rank the training examples by their scores and choose the topK example pairs. An overview of our methods is provided in Fig. 1(b).

In this work, we implement f θ as a combination of a neural network for feature extraction and the cosine distance function for measuring similarity between two feature vectors.

2.2.1 Unsupervised Prompt Retrieval

Our first method is unsupervised prompt retrieval where the key idea is to use an off-the-shelf feature extractor for extracting image features so that we can compare the cosine distance between the query x q and each training example x n ∈ D . In this case, the parameters θ for the score function f θ correspond to the off-the-shelf feature extractor, which are kept fixed.

2.2.2 Supervised Prompt Retrieval

The unsupervised method discussed above is not explicitly optimized for in-context learning; instead, it relies on how the feature extractor was pre-trained and the objective (function) used in pre-training may well not align with that of in-context learning. We propose a second method based on supervised prompt retrieval where we assume the source data contains labels. The goal is to directly optimize the score function f θ such that the chosen in-context example(s) can maximize the log-likelihood,

In this work, we present a simple implementation for the supervised method, which simply turns the unsupervised method into a supervised one by making the feature extractor learnable. In other words, we directly optimize Eq. 3 with respect to the feature extractor. Below we explain in detail how we train the feature extractor (see Fig. 2 for an overview).

̸

Data. Recall that we interpret the score f θ ( · , · ) as the cosine distance between two images in feature space. We would like to learn a space such that an image pair ( x n , x q ) with high in-context learning performance is close to each other, or far away from each other if the performance is low. Since there is no label defining how close a distance should be, we resort to contrastive learning for training the feature extractor. The goal is then to find a positive and a negative set for each training example x n ∈ D treated as a query. Specifically, for each example x n we compute the prediction ˆ y n = g τ (( x m , y m ) , x n ) where g τ is the large vision model defined in Sec. 2.1 and x m ∈ D but x m = x n . Since we have the ground truth y n for x n , we can measure the performance by comparing the prediction ˆ y n with the ground truth y n . Then, for each x n we choose the top-5 examples with the highest/lowest performance to form a positive/negative set.

Training. Let z n denote the features of x n extracted by the neural network we aim to optimize. At each iteraction, we sample a mini-batch B from the training dataset. Then, for each example in B , we sample one example from the top-5 positive and negative sets, respectively. The contrastive loss is computed as

where cos ( · , · ) is the cosine distance function, z + n denotes the feature representation of a positive example, and z -n denotes the feature representation of a negative example. It is worth noting that for mini-batch training, the negative set N contains a negative example of x n sampled from the top-5 negative set and other examples within the same mini-batch.

3 Experiments

In this section we conduct a comprehensive evaluation using different prompt selection methods (Sec. 3.1) and compare their robustness to distribution shifts (Sec. 3.2). We also provide extensive quantitative and qualitative analyses in Sec. 3.3 to help understand why our methods work and how to better apply them in practice. Source code will be released to the community for reproducing the full experiments.

Methods. All experiments are based on the image inpainting model pre-trained by [3] on a dataset consisting of academic figures. 3 This inpainting model comprises an encoder and a decoder, as depicted in Figure 2. The encoder generates a feature vector as its output, while the decoder yields a reconstructed image of the masked part of the input image. We mainly compare the following methods: (1) Random , the baseline method that randomly samples in-context examples from the source training dataset; (2) Unsupervised prompt retrieval (UnsupPR) , our first proposed method that uses off-the-shelf features for nearest example search. The main experiments are based on CLIP's vision encoder [16], which was pre-trained using multimodal contrastive learning; (3) Supervised prompt retrieval (SupPR) , our second proposed method that fine-tunes CLIP's vision encoder by directly optimizing in-context learning performance on downstream datasets. A variety of backbones are evaluated in Sec. 3.3.

Training details for the supervised model. The supervised model is trained for 200 epochs using SGD. The initial learning rate is set to 0.005, decayed by the cosine annealing rule. Each task need a specific supervised model for SupPR.

3.1 Main Results

Setup. Following [3], we evaluate our methods on three computer vision tasks, which have not been seen during the training of the image inpainting model. We provide the details about the datasets used for these tasks as follows. (1) Foreground segmentation : We use Pascal5 i [21], which has four non-overlapping splits each containing five categories. The results are averaged over all splits. (2) Single object detection : The experiments are done on Pascal VOC [7]. (3) Colorization : We use ImageNet-2012 [19], where the original validation set containing 50,000 images is used as our test set. The training data used to learn our supervised prompt retrieval model is created by randomly sampling 50,000 images from ImageNet's 1.2M training set. For all experiments, incontext examples come from the training set.

Results. Table 1 shows the results on the three benchmarks covering foreground segmentation, single object detection, and colorization. We summarize our findings as follows. First, prompt retrieval clearly outperforms random selection . In particular, the improvements of prompt retrieval over random selection are significant in foreground segmentation and single object detection: more than 6% on the former and 1% on the latter. However, the gains on colorization are only marginal (0.63 vs. 0.67), suggesting that the image inpainting model is probably weak at image coloriza-

tion. Second, the supervised prompt retrieval method performs the best . This is not surprising as the supervised method optimizes in-context learning performance concerning the prompt selection module. In contrast, the unsupervised method relies more on the off-the-shelf feature extractor. Overall, the results well justify the design of the prompt retrieval framework, which can serve as a strong baseline for future research.

3.2 Experiments on Distribution Shifts

Setup. Distribution shifts are commonly seen in real-world applications, and therefore AI models need to be robust to distribution shifts [24]. To test this ability in visual in-context learning, we create a new protocol focusing on foreground segmentation where the source dataset is Pascal while the target dataset is MSCOCO [12]. Specifically, we follow the design of Pascal5 i and create MSCOCO5 i , which also has four splits, each having the same set of categories as in the corresponding split in Pascal5 i . Note that such a shift mainly affects the supervised prompt retrieval method that requires training but not the unsupervised UnsupPR and Random.

Results. The results are shown in Table 2. First of all, the unsupervised prompt retrieval method beats the random selection method by a clear margin. By comparing the two prompt retrieval methods, we find that the supervised method again performs better than the unsupervised one despite being a learning-based approach-this is an exciting finding as it means the supervised method does not have the overfitting problem here. Nonetheless, we observe

that the gains achieved by the prompt retrieval methods here are generally smaller than the gains achieved on the standard foreground segmentation benchmark: here SupPR is only around 3% better on average than Random (19.95% vs.16.78%) while the improvement in Table 1 reaches 8% (35.56% vs. 27.56%). One potential solution to reduce the gap might be to improve the image inpainting model, which is beyond the scope of this paper.

3.3 Further Analysis

What are good in-context examples? To answer this question, we visualize the in-context examples found by UnsupPR and SupPR in Fig. 3. We focus on foreground segmentation and choose two categories from Pascal (person and cow). In each grid, the first row corresponds to the retrieved in-context example (i.e., an input-output pair) while the second row contains the query and model prediction. By comparing the in-context examples picked by UnsupPR and those picked by SupPR, we find the reason why SupPR performs better than UnsupPR: the examples found by SupPR are more similar to the queries in terms of semantics (e.g., Fig. 3(e)), background (e.g., Fig. 3(a)), object pose (e.g., Fig. 3(b), object appearance (e.g., Fig. 3(i)), viewpoint (e.g., Fig. 3(k)), and so on. We also observe similar patterns in other categories/tasks (please refer to the supplementary).

Backbone. To understand if using a different backbone than CLIP would make a big difference, we further evaluate our prompt retrieval methods, UnsupPR and SupPR, on the foreground segmentation benchmark using two other backbones: EVA [8] pre-trained using self-supervised learning (i.e., masked image modeling) and ViT [6] pre-trained using supervised learning. The results are reported in Table 3. Although these three backbones perform differently on image recognition under the fine-tuning setting-EVA performed the best-the gap between them

(g)

(h)

(i)

(j)

(k)

(l)

IoU: 37.31

IoU: 47.33

IoU: 42.33

IoU: 69.05

IoU: 26.13

IoU: 27.38

IoU: 58.59

IoU: 45.87

(a)

(b)

(c)

(d)

(e)

(f)

IoU: 49.05

IoU: 42.10

IoU: 66.93

IoU: 61.25

IoU: 74.44

IoU: 63.38

IoU: 36.67

IoU: 86.44

IoU: 86.64

IoU: 91.69

IoU: 80.14

IoU: 63.14

IoU: 79.22

IoU: 66.75

IoU: 49.03

v

v

v

for both UnsupPR and SupPR is less than 1%. Therefore, we can conclude that the backbone for visual in-context learning does not matter much.

Number of in-context examples. We follow [3] and create a large grid enough to fit 8 examples at maximum (as shown in Fig. 4 right). By varying the number of in-context examples from 1 to 7, we obtain a set of results and plot them in Fig. 4 left. Clearly, more in-context examples lead to better performance for all three methods, including SupPR, UnsupPR, and Random. This is probably because in-context examples can be viewed as 'training data', and having more training data typically benefits performance-in visual in-context learning, more training data gives a more comprehensive 'context.' We show a few example cases in Fig. 4 right to explain this observation.

IoU: 20.98

IoU: 15.68

IoU: 32.50

IoU:29.09

(a)

Num. of examples = 1

(b)

Num. of examples = 7

18.56

21.90

22.87

25.39

21.22

23.41

24.39

26.51

21.90

24.43

25.87

27.99

18

20

22

24

26

28

1

3

5

7

mIoU

Number of in-context

examples

Random

UnsupPR

SupPR

(c)

IoU:34.19

IoU:56.46

27.58

27.58

27.66

27.63

27.83

27.42

27.56

29.81

32.01

32.46

32.85

33.09

33.50

33.56

31.30

34.62

35.42

35.41

35.55

35.64

35.56

26

27

28

29

30

31

32

33

34

35

36

0.01

0.1

0.2

0.4

0.6

0.8

1

mIou

Size of retrieval set (% of full set)

Random

UnsupPR

SupPR

33.56

35.56

34.15

33.70

35.56

34.03

33.71

35.57

34.05

33

34

35

36

UnsupPR

SupPR

AVG

mIOU

Similarity metric selection

Cosine

Euclidean

Manhattan

Order of in-context examples. To understand if changing the order of in-context examples makes a difference, we fix the number of in-context examples to 3, evaluate all possible combinations, and compute the mean and standard deviation. As shown in Table 4, the standard deviation is generally small, so the order is not a concern as long as good examples are chosen.

Size of retrieval set. Recall that in-context examples are sampled from the training dataset, namely the retrieval set. We are interested to know whether the size has any impact on performance, especially for the supervised prompt retrieval method. To this end, we build seven subsets for each split in Pascal5 i , which cover a wide range of sizes (see the x-axis in Fig. 5 left). The results are plotted in Fig. 5 left. For random selection, the size does not matter at all. In contrast, the two prompt retrieval methods clearly benefit from a bigger size. But their performance plateaus when the size reaches a certain level. It is worth noting that for the supervised method, 20% of the total data is sufficient for achieving a decent performance.

SupPR vs. a supervised learning model.

To justify that the strength of SupPR comes from its design rather than the supervision signal, we further train a supervised learning model for each task and use the trained model to extract image features for prompt selection. This can be seen as replacing the off-the-shelf model in Un-

supPR with a supervised learning model. In this experiment, we fine-tune the state-of-the-art Masked Autoencoder (MAE) model [9] on the Pascal5 i dataset, and use the image encoder for feature extraction. Table 5 shows the results where SupPR largely outperforms the supervised MAE model.

Distance metric. We use the cosine distance by default to compute the score function in Eq. 2. Here we evaluate other design choices including Euclidean distance and Manhattan distance. As shown in Fig. 5 right, the results are very similar for different distance metrics.

4 Related Work

4.1 In-Context Learning

In-context learning is a novel learning-free paradigm that originally emerged in large language models, such as GPT-3 [5]. In computer vision, in-context learning is a relatively new concept. One of the earliest studies tackling in-context learning is Flamingo [2], a large visual language model that can be prompted with textual instructions and can cope with both images and video sequences. More relevant to our work is an image inpainting model developed by Bar et al. [3], which is pre-trained to fill missing patches in academic paper figures and capable of performing in-context learning on new tasks like foreground segmentation and image colorization. Another related work is Painter [22], a generalist model that is trained on the combination of a variety of computer vision tasks (e.g., depth estimation and semantic segmentation) using customized output spaces. Our work mainly follows the image inpainting model [3] where the pre-training task differs significantly from the downstream task, and reveals valuable insights about visual in-context learning.

4.2 Prompt Retrieval in NLP

The natural language processing community has found that the choice of in-context examples has a huge impact on performance [1, 13]. Moreover, the way how in-context examples are constructed can also affect performance, e.g., prompt length and order of in-context examples. These findings prompted the NLP community to study how to find good in-context examples for large language models, which largely inspired our research. Liu et al. [13] assumed that good in-context examples should be semantically close to query sentences, based on which they proposed to select nearest neighbors in the training set measured by a sentence encoder like RoBERTa [14]. Rubin et al. [18] first used an unsupervised method to retrieve some candidates, among which top examples were chosen using a supervised prompt retriever to maximize downstream performance. Different from the prompt retrieval research in NLP, our work focuses on large vision models, which are different from language models in many ways and require customized designs to deal with visual input.

5 Discussion and Conclusion

Our research presents a timely study on an emergent ability termed in-context learning for large vision models. We systematically investigate how the choice of in-context examples impacts downstream performance, exposing a critical issue that different in-context examples could lead to drastically different results. We then propose an effective prompt retrieval framework for visual in-context learning, with two simple implementations provided: one based on unsupervised learning and the other based on supervised learning. Our methods obtain significant improvements over random selection under various problem settings, demonstrating huge potential of prompt retrieval methods in black-box adaptation scenarios for vision foundation models.

Our research also unveils some intriguing phenomena. For instance, we show that a good in-context example should be semantically similar to the query and closer in context. A model that can better balance spatial and semantic closedness in feature space would be more ideal for visual in-context learning. We hope the insights presented in this work could pave the way for developing more effective prompt retrieval methods.

Our experiments show that our methods are not strong enough to cope with distribution shifts. Though our methods outperform random selection under distribution shifts, the gap is much smaller than that on a standard benchmark, suggesting huge room for improvement.

A Illustration of In-context Examples

In the supplementary material, we illustrate more in-context learning results of foreground segmentation, single object detection, and colorization tasks.

A.1 Foreground Segmentation

The main paper presents the in-context examples from the person and cow categories. In the supplementary, as shown in Fig. 6-11, we present examples from the remained 18 categories in Pascal-5 i .

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(r)

(s)

IoU: 67.02

IoU: 71.39

IoU: 38.79

IoU: 47.04

IoU: 27.84

IoU: 13.89

IoU: 52.09

(a)

(b)

(c)

(d)

(e)

(f)

IoU: 29.58

IoU: 34.12

IoU: 43.23

IoU: 74.64

IoU: 78.78

IoU: 43.13

IoU: 40.52

IoU: 35.02

IoU: 38.64

IoU: 16.38

IoU: 21.61

IoU: 19.90

IoU: 66.50

IoU: 54.57

IoU: 66.08

IoU: 57.10

IoU: 30.44

IoU: 44.75

IoU: 65.49

IoU: 54.25

IoU: 77.60

IoU: 81.50

IoU: 35.28

IoU: 66.75

IoU: 61.31

IoU: 33.79

REDRUM

LMPA

AHPA

A.2 Single Object Detection

As shown in Fig. 12-13, we illustrate the in-context examples from the single object detection task. By comparing the in-context examples picked by UnsupPR and those picked by SupPR, we find the examples found by SupPR are more similar to the queries in terms of object pose (e.g., Fig. 12(f)), viewpoint (e.g., Fig. 12(r).

A.3 Coloralization

As shown in Fig. 14-15, we illustrate the in-context examples from the colorization task. This task aims to map a gray-scale image to a color image. By comparing the in-context examples picked

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(r)

(s)

IoU: 21.72

IoU: 26.55

IoU: 13.89

IoU: 25.93

(a)

(b)

(c)

(d)

(e)

(f)

IoU: 11.12

IoU: 61.63

IoU: 43.06

IoU: 52.62

IoU: 62.24

IoU: 58.01

IoU: 28.07

IoU: 56.08

IoU: 17.95

IoU: 57.14

IoU: 38.60

IoU: 34.96

IoU: 71.44

IoU: 29.17

IoU: 70.87

IoU: 74.21

IoU: 14.59

IoU: 41.31

IoU: 64.74

IoU: 52.98

IoU: 63.29

IoU: 68.04

IoU: 72.49

IoU: 15.43

by UnsupPR and those picked by SupPR, we find the ground truth images of examples found by SupPR are more similar to that of the queries in terms of image style, e.g. the background color (e.g., Fig. 14(g)(h)).

(d)

IoU: 59.41

IoU: 67.96

IoU: 76.11

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(r)

(s)

IoU: 40.91

IoU: 10.53

IoU: 23.59

IoU: 28.56

IoU: 28.38

IoU: 42.44

(a)

(b)

(c)

(e)

(f)

IoU: 23.46

IoU: 33.00

IoU: 28.87

IoU: 34.29

IoU: 62.26

IoU: 50.00

IoU: 63.77

IoU: 63.17

IoU: 26.03

IoU: 84.00

IoU: 75.39

IoU: 79.47

IoU: 51.00

IoU: 48.62

IoU: 51.92

IoU: 72.58

IoU: 92.00

IoU: 57.00

IoU: 26.25

IoU: 36.67

(d)

IoU: 22.37

IoU: 65.46

IoU: 65.50

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(r)

(s)

IoU: 54.00

IoU: 68.04

IoU: 25.92

IoU: 21.99

IoU: 70.16

IoU: 38.34

(a)

(b)

(c)

(e)

(f)

IoU: 48.93

IoU: 73.70

IoU: 41.00

IoU: 54.02

IoU: 41.22

IoU: 60.00

IoU: 42.66

IoU: 26.76

IoU: 39.71

IoU: 67.09

IoU: 17.91

IoU: 22.38

IoU: 59.27

IoU: 71.61

IoU: 85.94

IoU: 49.44

IoU: 71.77

IoU: 68.91

IoU: 65.63

IoU: 60.82

IoU: 70.87

IoU: 72.97

IoU: 71.77

IoU: 51.60

IoU: 77.78

(d)

IoU: 63.09

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(r)

(s)

IoU: 31.88

IoU: 45.71

IoU: 34.58

IoU: 60.62

IoU: 27.09

(a)

(b)

(c)

(e)

(f)

IoU: 23.25

IoU: 43.73

IoU: 27.15

IoU: 29.16

IoU: 56.38

IoU: 28.82

IoU: 62.40

IoU: 41.35

IoU: 93.05

IoU: 39.02

IoU: 45.35

IoU: 65.69

IoU: 18.00

IoU: 46.55

IoU: 33.91

IoU: 35.21

IoU: 33.75

IoU: 47.07

IoU: 41.56

IoU: 38.61

(d)

IoU: 10.59

IoU: 33.28

IoU: 40.88

(g)

(h)

(i)

(j)

(k)

(l)

(a)

(b)

(c)

(e)

(f)

IoU: 18.61

IoU: 26.82

IoU: 25.88

IoU: 44.57

IoU: 33.40

IoU: 61.47

IoU: 51.28

IoU: 59.97

IoU: 68.02

IoU: 66.10

IoU: 32.34

IoU: 30.13

IoU: 54.34

IoU: 27.73

IoU: 20.43

IoU: 21.28

IoU: 11.90

IoU: 36.78

IoU: 71.30

IoU: 51.64

IoU: 27.65

IoU: 61.60

IoU: 32.60

IoU: 58.09

IoU: 22.71

IoU: 23.47

IoU: 68.81

(m)

(n)

(o)

(p)

(r)

(s)

(d)

IoU: 31.79

IoU: 32.98

IoU: 63.17

(g)

(h)

(i)

(j)

(k)

(l)

(a)

(b)

(c)

(e)

(f)

IoU: 36.21

IoU: 13.14

IoU: 15.34

IoU: 6.89

IoU: 39.50

IoU: 14.15

IoU: 73.74

IoU: 29.59

IoU: 67.14

IoU: 67.26

IoU: 69.75

IoU: 32.34

IoU: 53.40

IoU: 71.97

IoU: 52.23

IoU: 26.33

IoU: 20.43

IoU: 20.72

IoU: 41.23

IoU: 71.30

IoU: 29.61

IoU: 42.64

IoU: 45.97

IoU: 45.88

IoU: 24.76

(m)

(n)

(o)

(p)

(r)

(s)

c

BAGGAGE

mse : 0.79

mse : 0.88

mse : 1.13

mse: 0.91

mse : 0.85

mse : 0.63

(d)

mse : 0.49

(a)

(b)

(c)

(e)

(f)

mse : 0.53

mse : 0.67

mse : 0.52

mse : 0.78

mse :

0.51

mse : 1.15

mse : 0.68

mse : 0.76

mse : 2.16

mse : 0.99

mse : 0.38

(g)

(h)

(i)

(j)

(k)

(l)

mse : 0.88

mse : 0.73

mse : 0.23

mse : 0.17

mse : 0.40

mse : 0.48

mse : 0.66

mse : 1.15

mse : 1.51

mse: 1.01

mse : 0.78

mse : 0.79

(d)

mse : 0.44

(a)

(b)

(c)

(e)

(f)

mse : 0.80

mse : 0.64

mse : 0.80

mse : 0.55

mse : 0.51

mse : 3.48

mse : 1.19

mse : 0.79

mse : 0.88

mse : 0.80

mse : 0.99

(g)

(h)

(i)

(j)

(k)

(l)

mse : 0.36

mse : 0.52

mse : 0.85

mse : 0.73

mse : 0.49

mse : 0.34

m

References

Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In-context examples selection for machine translation. arXiv preprint arXiv:2212.02437 , 2022. 9

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022. 9

Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems (NeurIPS) , 35:25005-25017, 2022. 2, 3, 5, 7, 9

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021. 1

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS) , 33:1877-1901, 2020. 3, 9

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. 6

Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision (ICCV) , 111(1):98-136, 2015. 5

Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636 , 2022. 6

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16000-16009, 2022. 9

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 , 2021. 1

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021. 1

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision (ECCV) , pages 740-755. Springer, 2014. 6

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt3 ? arXiv preprint arXiv:2101.06804 , 2021. 9

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019. 9

Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image-to-video transfer learning for action recognition. arXiv preprint arXiv:2206.13559 , 2022. 1

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , pages 8748-8763. PMLR, 2021. 1, 2, 5

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning (ICML) , pages 8821-8831. PMLR, 2021. 1

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633 , 2021. 9

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li FeiFei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. 5

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022. 1

Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. british machine vision conference (BMVC) , 2017. 2, 5

Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499 , 2022. 9

Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673 , 2022. 1

Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2022. 6

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 1

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV) , 2022. 1

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied in-context learning abilities of large vision models and finds downstream task performance to be highly sensitive to the choice of examples. They observe that the closer the in-context example is to the query, the better the performance. Since manually designing prompts would be time-intensive, they propose a (contrastive) supervised and an unsupervised version of a framework for prompt retrieval guided by a score function. They evaluate the proposed framework on the foreground segmentation, single object detection and image colorisation tasks.\n\nSTRENGTHS:\n- In-context learning in vision is a new and active area of research and methods for selecting better examples for in-context learning will likely be of interest to the computer vision community. \n- The proposed method for prompt retrieval outperforms randomly selected examples on all tasks considered.\n- Distribution shift results are interesting, suggesting that the supervised prompt retrieval method acquires in-context knowledge that is robust to distribution shift.\n- The analysis and ablations are interesting and insightful.\n\nWEAKNESSES:\n- The finding that examples in the prompt should be semantically close to the test example is known from language and not that surprising.\n- The proposed method is technically relatively straightforward compared to an average NeurIPS paper.\n- The analysis is limited to one model for in-context learning and may not transfer to other models.\n\nQUESTIONS:\n1. It is surprising to me that the smallest gains are observed on the colorisation task and supervision is not helpful, despite the fact that both training and evaluation are done on ImageNet. Do you have any intuition for why this may be the case?\n2. In Sec 3.3, you mention that 20% of the data is enough for good performance. Is this 20% of 50,000, i.e. 1000 images?\n3. What is the intuition behind MAE in Table 5 performing worse than UnsupPR?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies that visual prompting is sensitive to the choice of input-output example(s). To address this, the authors propose a retrieval framework to better select the examples. The authors propose supervised and unsupervised retrieval approaches which significantly improve the performance compared to random selection.\n\nSTRENGTHS:\n- Through extensive empirical study, the authors demonstrate the role of input-output examples for visual prompting. \n- The authors present two different retrieval strategies to choose the best visual prompting example and find that both approaches are superior to random choice.\n- The authors show that choosing the right example can also improve performance under distribution shifts. \n- It is also nice that the retrieval similarity function is class-agnostic. But regardless, empirically it extends to three different tasks.\n\nWEAKNESSES:\nOverall, the main two weaknesses in my opinion are:\n1. The assumption that we have a set of tagged examples to retrieve from.\n2. The supervised/unsupervised similarity function is straightforward but currently very specific to mIOU similarity.\n\nMinor:\n\n3. I think there is a broader spectrum that can be analyzed for visual prompts.\n\nPlease see the questions for extended comments/feedback.\n\nQUESTIONS:\n1. The main weakness is the assumption that we have a set of tagged examples. In this case, why should we retrieve rather than use our tagged examples? e.g., train a new model overall tagged examples or use an ensemble? Perhaps the best way to address this would be to add another ensemble/supervised-trained baseline in Figure 5 (left). These baselines should utilize the same pool of examples.\n\n2. The supervised/unsupervised similarity function is straightforward but currently very specific to mIOU similarity. Can we improve this by utilizing similarity in feature space? (e.g., using existing CLIP or DINO, etc?). To be fair, from the empirical results it seems like by optimizing mIOU there is somewhat improvement for colorization and single object detection as well.\n\nMinor:\n\n3. I think there is a broader spectrum that can be analyzed for visual prompts. E.g., using synthetic task data as examples, using examples from different classes, then examples from within classes as analyzed in the paper.\n\nLIMITATIONS:\nThe authors clearly stated and discussed the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper investigates in-context learning for large vision models. Specifically, the authors propose to automatically retrieve prompt for vision models in two methods: (1) nearest example search with off-the-shelf model, (2) supervised prompt retrieval method.
Experimental results show that the proposed method bring non-trivial improvement, comparing to the random selection on foreground segmentation, single object detection and colorization.

STRENGTHS:
- the motivation of this paper is important: investigating how vision models could benefit from in-context learning.

- the writing is clear and easy to follow.

- The experiments show improvement comparing with random selection.

WEAKNESSES:
1. Visual in-context learning emerged from large autoregressive language models. In-context learning was referred to use language models for a wide range of downstream tasks without updating the model itself. But this paper, though claims to study visual in-context learning, only studies a limited set of tasks (assign labels to pixels) with in-paining model. I wonder if the authors could provide thoughts on how to apply to a wider visual tasks, like classification, or standard segmentation tasks (semantic and instance).


2. The finding of "a good in-context example should be semantically similar to query and closer in context" seems to be a problem if the testing examples are unique, and there is no semantically similar {image, annotation} pair in the retrieval pool. This goes to the practicality of the method, and goes to the claim "potential of using prompt retrieval in vision applications".

QUESTIONS:
please refer to the weakness section

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the problem of vision in-context-learning that the performance highly depends on the choice of visual in-context examples. In the paper, the authors propose automatically retrieving prompts in unsupervised and supervised ways without reaching internal weights of large vision models. Besides, this paper comprehensively studies how to select good examples for visual in-context learning and shares some valuable insights with the community on choosing good visual in-context examples.\n\nSTRENGTHS:\nThe paper discusses several aspects that influence visual in-context learning and propose methods to choose in-context learning samples automatically to optimize visual in-context learning with inpainting method. The proposed methods both outperform random selection baseline and the supervised version requires training performs better than the unsupervised method in several tasks. Also, this paper discusses other factors that might influence visual in-context-learning including the number of examples, order of examples, and size of retrieval set, which shares some useful practical experience to the community.\n\nWEAKNESSES:\n1. This paper only discusses the previous paper of using inpainting as visual in-context learning as the visual icl frame and experiment on the dataset from that paper. It's hard to tell if experiments and conclusions conducted on a specific framework will generalize to more general \"visual in-context learning\" settings. Besides, although the supervised example retrieval method outperforms the unsupervised one, the additional model and training seem to be contradictory to the main advantages of ICL learning that requires no additional training.\n\nQUESTIONS:\n1. The paper discusses the impact of the order and number of in-context examples. While more in-context examples generally lead to better performance, is there an upper limit to this? Will adding more examples become redundant or even detrimental at some points?\n\n2. The paper compares the proposed methods with random selection. Are there other baselines that the authors considered for comparison?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}