What Makes Good Examples for Visual In-Context Learning?
Authors: Yuanhan Zhang
Kaiyang Zhou
Ziwei Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: computer vision
visual in-context learning
prompt learning

Abstract: Large vision models with billions of parameters and trained on broad data have great potential in numerous downstream applications. However, these models are typically difficult to adapt due to their large parameter size and sometimes lack of accesss to their weights---entities able to develop large vision models often provide APIs only. In this paper, we study how to better utilize large vision models through the lens of in-context learning, a concept that has been well-known in natural language processing but has only been studied very recently in computer vision. In-context learning refers to the ability to perform inference on tasks never seen during training by simply conditioning on in-context examples (i.e., input-output pairs) without updating any internal model parameters. To demystify in-context learning in computer vision, we conduct an extensive research and identify a critical problem: downstream performance is highly sensitivie to the choice of visual in-context examples. To address this problem, we propose a prompt retrieval framework specifically for large vision models, allowing the selection of in-context examples to be fully automated. Concretely, we provide two implementations: (i) an unsupervised prompt retrieval method based on nearest example search using an off-the-shelf model, and (ii) a supervised prompt retrieval method, which trains a neural network to choose examples that directly maximize in-context learning performance. Both methods do not require access to the internal weights of large vision models. Our results demonstrate that our methods can bring non-trivial improvements to visual in-context learning in comparison to the commonly-used random selection. Code and models will be released.

What Makes Good Examples for Visual In-Context Learning?
Yuanhan Zhang' Kaiyang Zhou² Ziwei Liu S-Lab, Nanyang Technological University, Singapore Hong Kong Baptist University, Hong Kong (yuanhanC02, ziwei, ....1ii. Liue@ntu. edu sg kyzhou8hkbu.c edu.
Abstract
Large vision models with billions of parameters and trained on broad data have great potential in numerous downstream applications. However, these models are typically difficult to adapt due to their large parameter size and sometimes lack of accesss to their weights-entities able to develop large vision models often provide APIS only. In this paper, we study how to better utilize large vision mod- els through the lens of in-context learning, concept that has been well-known in natural language processing but has only been studied very recently in com- puter vision. In-context learning refers the ability to perform inference tasks never seen during training by simply conditioning on in-context examples (i.e., input-output pairs) without updating any internal model parameters. To de- mystify in-context learning in computer vision, we conduct an extensive research and identify critical problem: downstream performance highly sensitivie the choice of visual in-context examples. To address this problem, we propose a prompt retrieval framework specifically for large vision models, allowing the se- lection of in-context examples to be fully automated. Concretely, we provide two implementations: (i) an unsupervised prompt retrieval method based on nearest example search using off ofi-tf model, and (ii) supervised prompt retrieval method, which trains neural network to choose examples that directly maximize in-context learning performance. Both methods do not require access to the inter- nal weights of large vision models. Our results demonstrate that our methods can bring non-trivial improvements visual in-context learning in comparison to the commonly-ure used random selection.
Introduction
recent years, large-scale models have emerged in computer vision: they have enormous parameter sizes and pre-trained on broad data gain wide-ranging knowledge. These models have demon- strated remarkable generalization performance and have great potential for numerous downstream applications [4]. However, due the large model size and the potentially proprictary data used for training, entities able to develop large-scale models typically only provide users with APIs, known ---eevice (Maas). Representative examples include the prominent text-to-image gen- eration models, DALL-E [17] and Imagen [20], and OpenAl's powerful language models like GPT- unable apply full fine-tuning some parameter-efficient
3/ChatGPT [16]. As result, tuning techniques, such as prompt learning [11, 10, 26, 25, 23, 15], for model adaptation, largely limiting downstream performance. To cireumvent the problem, "black-box" adaptation methods need to be developed.
Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Foreground segmentation (mloU)
Average Standard deviation
Database (train)
Query (test)
100
Score
function
fg(
Retrieved image
Query imaee indecc
prompt
Worst prompt
Large-scale
Ourpot
vision model
(a) Visual in-context learning sensitive to prompt selection
(b) Promip for visual in-context learning
Figure I: (a) Different choices of in-context examples (outlined in green) often lead significantly different results. Here we show 30 random query images (x-axis) from Pascal-5' [21] split O, and measure the performance range using 50 different in-context examples. (b) We propose prompt retrieval framework aiming to automate the selection of in-context examples. We provide two im- plementations of the idea: one is unsupervised while the other is supervised, both outperforming random selection by a clear margin and without accessing the large vision model's internal weights.
In-context learning, which is a "hidden" capability originally found in large autoregressive language models [16], has recently been investigated for large vision models [3], and more importantly, has the potential to be adopted as the mainstream approach in MaaS applications in the future. Without the need to update any parameter when deploying previously unseen tasks, in-context learning input-output pairs, called in-context examples or prompt, test example, which together guide the model to produce an ideal result. For instance, in natural language processing one could prepend a French- French-English sentence pair to a French sentence, and model would produce an English translation of the French sentence. In computer vision, Bar et al. [3] pre-trained neural network to fill missing patches in grid-like images, which allows the model perform in-context learning for unseen tasks like image segmentation (see the grid images in Fig. ((a) bottom).
simply prepends some
In this work, we focus on visual in-context learning, a relatively new concept with little existing research regarding how to better apply it in practice. We for the first time conduct comprehensive investigation on the impact of in-context examples for large vision models, and identify a critical issue: downstream performance is highly sensitive to the choice of in-context examples. This is evidenced by the large variances observed for a variety of test examples shown in Fig. ((a) top. By visualizing the results in Fig. 1(a) bottom, it seems to suggest that the closer the in-context example to the query, the better the result. For example, the best prompt image closer to the query as they are similar in object pose and background; on the other hand, the worst prompt image has drastically different style than the query image, which might explain why the predicted mask focuses on the wrong region, i.c., the white pillar instead of the cat.
Clearly, designing proper prompt containing the optimal in-context example(s) by hand would be extremely difficult. To overcome the problem, we propose prompt retrieval framework where the core component is a score function, which aims to give each source instance a score to indicate the level of suitability for being included in the prompt. Once the scoring process is done, we can simply pick one or multiple examples with the highest score(s) to construct a prompt. An overview of our framework is depicted in Fig. ((b).
We provide two implementations for the prompt retrieval framework, both interpreting the score as the cosine distance measuring similarity between a query and a source example. The first is an unsupervised method based on nearest example search using an off-the-shelf model. The second
"To clarify, in natural language processing, prompt refers to the instruction input specifying particular purpose while in context examples special prompt, which consist of Nomain-specitte ouputt pairs.

---

Promat
Posttres: Exemples with high
Trery
Query
I (-mpainted)
Randomly sampled prompt candidates
Feature space
(train)
Negatives: Examples with lew IoU
Large-scale
result of ququrry
visian model
Learnable feature extractor
Query (train)
Figure 2: Overview of the supervised prompt retrieval method. The main idea to compute the in-context learning result for each source example, and pick those with the highest/lowest results to form a positive/negative set for contrastive learning.
is supervised method, which learns neural network to choose examples that directly maximize in-context learning performance. Since there is no ground-truth score to be used as supervisory signal, we resort to contrastive learning paradigm: source examples that result in better (or worse) in-context learning performance should get closer (or farther) to the query in feature space.
Our contributions and the main findings are summarized as follows. (1) We present the first compre- hensive study concerning how select good examples for the emerging visual in-context learning, and reveal a critical issue that the choice of in-context examples has huge impact on performance. (2) From the technical perspective, we present prompt retrieval framework that can automate the prompt selection process, and provide two simple implementations:
unsupervised method and a supervised method. (3) By conducting extensive experiments on three visual in-context learn- ing tasks (which have not been seen during pre-training), namely foreground segmentation, sin- gle object detection and image colorization, we share valuable insights with the community on how find good visual in-context examples, e.g., the supervised method performs the best and often finds examples that are both semantically close and spatially similar to a query. (4) The code and models are available https: ////ttttttu com/ Zhang AI AI/visual. prompt_retries
Methods
2.1 Visual In-Context Learning
In-context learning is a new paradigm that originally emerged from large autoregressive language models pre-trained on broad data, such GPT-3 [5]. Unlike traditional learning methods, in-context learning does not require any parameter update and instead conditions prediction on some in-context examples in the form of input-output pairs. For example, in natural language processing one might give a French- -nngish sentence pair and a test French sentence as input to the model, which then produces the English version of the sentence. In computer vision, such paradigm has only been studied very recently. For example, Bar el al. [3] trained a neural network to fill missing patches in grid-like images, which in allows the model to perform in-context learning on unseen tasks.
Formally, given dataset = {In, yn) b containing image-label pairs (c.g., an image and its segmentation mask), query example Eq. and model gr, in-context learning be formulated as:
y4 9- (P,
(I)
where P is called a prompt, which consists of K input-output pairs, P = {Key a Me se...... Teg: Yeg C D. In particular, the prompt P provides some context for guiding the model to produce the ideal Yg for n. without updating the large model's parameters т.
Problem.
The most common approach for designing the prompt P in the vision domain is (within- class) random selection proposed by [3]: one or multiple image-label pairs (with the same label the test example) are randomly chosen from the training dataset. As illustrated in Fig. 1(a), the
3

---

performance is highly sensitive to the selection of in-context examples the gap between the best and worst prompt could reach over 70% mloU. Below we propose two automatic prompt selection methods to tackle this problem.
2.2 Prompt Retrieval
Our goal is to automatically select the most suitable example(s) from the training dataset for query Tg.. To this end, we propose prompt retrieval framework in the following form,
arg max [o( Tg). ED
(2)
where fo is a function parameterized by A, aiming to produce a score for a pair of v,, and zg. When K 1, choose the optimal example pair as the prompt, P fa+ }. When K > 1, rank the training examples by their scores and choose the top- example pairs. An overview of our methods is provided in Fig. (b).
In this work, we implement fe as combination of neural network for feature extraction and the cosine distance function for measuring similarity between two feature vectors.
2.2.1 Unsupervised Prompt Retrieval
Our first method is unsupervised prompt retrieval where the key idea is off-the-shelf feature extractor for extracting image features so that we can compare the cosine distance between the query Hy and each training example In E D. this case, the parameters for the score function fo correspond to the off-the-shelf feature extractor, which are kept fixed.
2.2.2 Supervised Prompt Retrieval
The unsupervised method discussed above is not explicitly optimized for in-context learning; in- stead, it relies on how the feature extractor was pre-trained and the objective (function) used in pre-training may well not align with that of in-context learning. We propose second method based on supervised prompt retrieval where we assume the source data contains labels. The goal is to directly optimize the score function fe such that the chosen in-context example(s) can maximize the log-likelihood,
max P
log log x.)))
(3)
In this work, we present a simple implementation for the supervised method, which simply turns the unsupervised method into supervised one by making the feature extractor learnable. In other words, we directly optimize Eq. 3 with respect to the feature extractor. Below we explain in detail how we train the feature extractor (see Fig. 2 for an overview).
Data. Recall that we interpret the score fo(;,) the cosine distance between two images in feature space. We would like to learn a space such that an image pair (I,, F.g) with high in-context learning performance close each other, far away from each other if the performance is low. Since there is no label defining how close a distance should be, we resort to contrastive learning for training the feature extractor. The goal is then to find a positive and negative set for each training example En E 'D treated as a query. Specifically, for each example I, we compute the prediction d. 9r (=(( Vn),, xx,) where 9- is the large vision model defined in Sec. 2.1 and I'm Since we have the ground truth yn. for In, we can measure the performance by comparing the prediction Ón with the ground truth y.. Then, for each Fn we choose the top-5 examples with the highest/lowest performance to form a positive/negative set.
but Z'm o In-
Training. Let 2, denote the features of F, extracted by the neural network aim to optimize. At each iteraction, we sample a mini-batch B from the training dataset. Then, for each example in B3 sample one example from the top-5 positive and negative sets, respectively. The contrastive loss is computed as
Σ B
(cos(3n12n) ) + Σ ccos(zn;20) log
(4)
B
An EN

---

Table 1: Main results. The prompt retrieval methods outperform random selection, and the supervised method achieves the best performance.
Seg. (mloU)
Det. (mloU)
Color. (msc) +
Split-O
Split-1
Split-2
3plit-
Avg
Random 28.66
30.21
27.81
23.55
27.56
25.45
0.67
UnsupPR
34.75
35.92
32.41
31.16
33.56
26.84
0.63
SupPR
37.08
38.43
34.40
32.32
35.56
28.22
0.63
where cos(:;) :) the cosine distance function, =, denotes the feature representation of positive example, and 2,, denotes the feature representation of negative example. It is worth noting that for mini-batch training, the negative set contains negative example of 2n sampled from the top-5 negative set and other examples within the same mini-batch.
3 Experiments
In this section we conduct comprehensive evaluation using different prompt selection methods (Sec. 3.1) and compare their robustness to distribution shifts (Sec. 3.2). We also provide extensive quantitative and qualitative analyses in Sec. 3.3 help understand why methods work and how to better apply them in practice. Source code will be released to the community for reproducing the full experiments.
Methods. All experiments are based on the image inpainting model pre-trained by [3] on a dataset consisting of academic figures, This inpainting model comprises an encoder and decoder, as de- picted Figure 2. The encoder generates feature vector as its output, while the decoder yields a reconstructed image of the masked part of the input image. We mainly compare the following meth- ods: (1) Random, the baseline method that randomly samples in-context examples from the source training dataset; (2) Unsupervised prompt retrieval (UnsupPR), our first proposed method that uses off-the-shelf features for nearest example search. The main experiments are based on CLIP's vision encoder [16], which was pre-trained using multimodal contrastive learning; (3) Supervised prompt retrieval (SupPR), our second proposed method that fine-tunes CLIP's vision encoder by directly optimizing in-context learning performance downstream datasets. A variety of backbones evaluated in Sec. 3.3.
Training details for the supervised model. The supervised model is trained for 200 epochs using SGD. The initial learning rate is set 0.005, decayed by the cosine annealing rule. Each task need specific supervised model for SupPR.
3.1 Main Results
Setup. Following [3], we evaluate our methods on three computer vision tasks, which have not been seen during the training of the image inpainting model. We provide the details about the datasets used for these tasks as follows. (1) Foreground segmentation: We use Pascal-5' [21], which has four non-overlapping splits each containing five categories. The results are averaged over all splits. (2) Single object detection: The experiments are done on Pascal VOC [7]. (3) Colorization: We use ImageNet-2011 [19], where the original validation set containing 50,000 images is used as our test set. The training data used to learn our supervised prompt retrieval model is created
by randomly sampling 50,000 images from ImageNet's 1.2M training set. For all experiments, in- context examples come from the training set.
Results. Table shows the results on the three benchmarks covering foreground segmentation, single object detection, and colorization. We summarize our findings as follows. First, prompt re- trieval clearly outperforms random selection. In particular, the improvements of prompt retrieval over random selection are significant in foreground segmentation and single object detection: more than 6% on the former and 1% on the latter. However, the gains colorization are only marginal (0.63 vs. 0.67), suggesting that the image inpainting model probably weak image coloriza-
Phttps: ://:::/ggggtttt gtthhb....b/tgggggg comm/mmiroar/viiuee promptinggpggggggggggggg
5

---

tion. Second, the supervised prompt retrieval method performs the best. This is not surprising as the supervised method optimizes in-context learning performance conceriing the prompt selection module. In contrast, the unsupervised method relies more on the off-the-shelf feature extractor. Overall, the results well justify the design of the prompt retrieval framework, which can serve as a strong baseline for future research.
3.2 Experiments on Distribution Shifts
Setup. Distribution shifts are commonly seen in real-world applications, and therefore Al mod- els need be robust distribution shifts [24]. To test this ability in visual in-context learning, we create new protocol focusing on foreground segmentation where the source dataset is Pascal while the target dataset is MSCOCO [12]. Specifically, we follow the design of Pascal-5' and create MSCOCO-5', which also has four splits, each having the same set of categories as in the corre- sponding split in Pascal-5'. Note that such a shift mainly affects the supervised prompt retrieval method that requires training but not the unsupervised UnsupPR and Random.
Results. The results shown in Table 2. First of all, the unsupervised prompt re- trieval method beats the random selection method by a clear margin. By comparing the two prompt retrieval methods, we find that the supervised method again performs better than the unsupervised one despite being learning-basd based approach- thiraachhhhhst is an exciting finding as it means the super- vised method does not have the overfitting problem here. Nonetheless, we observe that the gains achieved by the prompt retrieval methods here are generally smaller than the gains achieved on the standard foreground segmentation benchmark: here SupPR is only around 3% bet- ter on average than Random (19.95% vs. 16.7%%) while the improvement in Table reaches 8% (35.56% vs. 27.56%). One potential solution to reduce the gap might be to improve the image inpainting model, which is beyond the scope of this paper.
Table 2: Results on distribution shifts (from Pascal to MSCOCO), Despite being a learning- approach, SupPR shows stronger robustness than UnsupPR and Random, which do not require any training.
Seg. (mloU) f
Split-O
Split-1
Split-2
Split-3
Avg
Random UnsupPR
12.17
18.47
20.55
15.94
16.78
12.67
19.62
21.33
18.44
18.02
SupPR 13.62
21.25
24.46
20.44
19.95
3.3 Further Analysis
What are good in-context examples? To answer this question, we visualize the in-context exam- ples found by UnsupPR and SupPR in Fig. We focus on foreground segmentation and choose two categories from Pascal (person and cow). In each grid, the first row corresponds to the retrieved in-context example (i.e., an input-output pair) while the second row contains the query and model prediction. By comparing the in-context examples picked by UnsupPR and those picked by SupPR, we find the reason why SupPR performs better than UnsupPR: the examples found by SupPR are more similar to the queries in terms of semantics (e.g., Fig. 3(e)). background (e.g.. Fig. 3(a)), object pose (e.g., Fig. 3(b), object appearance (e.g., Fig. 3()), viewpoint (e.g., Fig. 3(k)), and so on. We also observe similar patterns in other categories/tasks (please refer to the supplementary).
Backbone. To understand if using dif- ferent backbone than CLIP would make big difference, further evaluate our prompt retrieval methods, UnsupPR and SupPR, on the foreground segmentation benchmark using two other backbones: EVA [8] pre-trained using self-supervised learning (i.e., masked image modeling) and ViT [6] pre-trained using supervised learning. The results are reported in Ta- ble 3. Although these three backbones perform differently on image recognition under the fine-tuning setting-EVA per- formed the best-the gap between them
Table 3: Comparison between different backbones pre-trained using different methods: multimodal con- trastive learning for CLIP, self-supervised learning for EVA, and supervised learning for VIT. Overall, the per- formance is insensitive to the choice of backbones.
Seg. (mloU) f
Split-O Split-1 Split-2 Split-3
Avg
CLIP
34.75
35.92
32.41
31.16
33.56
UnsupPR
EVA
34.75
36.09
32.11
31.61
33.64
VIT
35.10
37.37
32.05
30.80
33.83
CLIP
37.08
38.43
34.40
32.32
35.56
SupPR
EVA
36.11
39.14
34.31
33.30
35.71
VIT
36.80
39.70
34.71
33.25
36.12

---

loU: 74-44
IoU: 61.25
loU: 63.38
IoU: 66.93
IoU: 36.67
IoU: 8.45
loU: 91.69 (a)
IoU: 86.64 (b)
loU: 80.14
IoU: 86.44 (d)
79.22 (e)
loU: 63.14
IoU: 26.13
IoU: 42.33
loU: 27.38
IoU:
IoU: 42.30
loU: 49.05
IoU: 58.59 (g)
IoU: 69.05
loU: 45.87
IoU: (
loU: 49.03 (k)
Iol: 66.75
Figure 3: In-context examples retrieved by UnsupPR and SupPR. In each grid, the first row contains prompt while the second row contains the query and prediction. The in-context examples found by SupPR are more similar than those found by UnsupPR to the queries in numer of ways: seman- tics (c.g., (c)), background (e.g.. (a)), object pose (e.g.. (b), object appearance (c.g., (i)), viewpoint (e.g.. (k)), etc. More examples can be found in the supplementary.
Table 4: Impact of the order of in-context examples. The results of different orders are averaged and their standard deviations are computed. The small variations in the results suggest that the order of in-context examples does not matter.
Seg. (mloU) Split-2
Split-O
Split-1
Split-3
Avg
Random
17.93 0.20
25.48 = 0.27
21.34 = 0.73
21.12 # 0.53
21.46 = 0.43
UnsupPR
20.22 0.31
27.58 0.40
22.42 4 0.38
23.36 0.42
23.39 + 0.37
SupPR
20.74
28.19
23.09
24.22 0.48
24.06
for both UnsupPR and SupPR is less than 1%. Therefore, conclude that the backbone for visual in-context learning does not matter much.
Number of in-context examples. We follow [3] and create a large grid enough to fit 8 examples maximum (as shown in Fig. right). By varying the number in-context examples from 1 to 7, we obtain set of results and plot them in Fig. 4 left. Clearly, more in-context examples lead to better performance for all three methods, including SupPR, UnsupPR, and Random. This probably because in-context examples can be viewed as "training data", and having more training data typically benefits performance- visual in-context learning, more training data gives a more comprehensive "context." We show few example cases in Fig. 4 right to explain this observation.

---

28
Num. of exeeampplee
24.36
mlou
1140
loU: 20.g8
loU: 15.68
1oU:34.19
Num of Cesamppeee
18
Number of cont-nttett
examples
IoU: 32.50 (a)
loU:2g.0g
loU:56.46
- Random
- UnsupPR
- SupPR
(b)
(c)
Figure 4: (Left) Impact of the number of in-context examples. (Right) More in-context examples can lead to better performance. The query in each grid is shown in the bottom right.
35.41
35.55
35.64
35.56
36
32:01
12.66
32.65
33.09
33.50
33.56
35.5
31.30
35
34.15
27.58
27.58
27.42
27.56
26
35
01
0.2
8.4
0.6
68
UnsupPR
AVG
retrieval set set)
Similarity metric selection
Random
-n-UnsupPR
SupPR
Cosine
B Euclidean
Manhattan
Figure 5: (Left) Impact the size of retrieval set. (Right) Ablation study on distance metric used compute the score function in Eq. 2. It can be observed that different metrics perform similarly.
Order of in-context examples. To understand if changing the order of in-context examples makes difference, we fix the number of in-context examples to 3, evaluate all possible combinations, and compute the mean and standard deviation. As shown in Table 4, the standard deviation generally small, the order is not concern long good examples chosen.
Size of retrieval set. Recall that in-context examples are sampled from the training dataset, namely the retrieval set. We are interested to know whether the size has any impact on performance, espe- cially for the supervised prompt retrieval method. To this end, we build seven subsets for each split in Pascal-5', which cover wide range of sizes (see the x-axis in Fig. 5 left). The results plotted in Fig. 5 left. For random selection, the size does not matter at all. contrast, the two prompt retrieval methods clearly benefit from bigger size. But their performance plateaus when the size reaches a certain level. It is worth noting that for the supervised method, 20% of the total data is sufficient for achieving decent performance.
SupPR vs. supervised learning model. To justify that the strength of SupPR comes from design rather than the su- pervision signal, we further train a super- vised learning model for each task and use the trained model to extract image features for prompt selection. This can be seen as replacing the off-the-shelf model in Un-
Table 5: Comparison between SupPR and fine-tuned MAE.
Seg. (mloU) f
Split-O
Split-1
Split-2
Split-3
Avg
MAE SupPR
31.69
34.26
30.01
28.55
31.27
37.08
38.43
34.40
32.32
35.56

---

supPR with a supervised learning model. In this experiment, we fine-tune the Sate-of-the-an Masked Autoencoder (MAE) model [9] on the Pascal-5' dataset, and use the image encoder for feature extraction. Table 5 shows the results where SupPR largely outperforms the supervised MAE model.
Distance metric. We use the cosine distance by default to compute the score function in Eq. 2. Here evaluate other design choices including Euclidean distance and Manhattan distance. As shown in Fig. right, the results are very similar for different distance metrics.
4 Related Work
4.1 In-Context Learning
In-context learning a novel learning-frre paradigm that originally emerged in large language mod- els, such as GPT-3 [5]. In computer vision, in-context learning is a relatively new concept. One of the earliest studies tackling in-context learning is Flamingo [2], large visual language model that can be prompted with textual instructions and can cope with both images and video sequences. More relevant to our work is an image inpainting model developed by Bar et al. [3], which is pre-trained to fill missing patches in academic paper figures and capable of performing in-context learning new tasks like foreground segmentation and image colorization. Another related work is Painter [22], a generalist model that is trained on the combination of variety of computer vision tasks (e.g., depth estimation and semantic segmentation) using customized output spaces. Our work mainly follows the image impainting model [3] where the pre-traini rre-training task differs significantly from the downstream task, and reveals valuable insights about visual in-context learning.
4.2 Prompt Retrieval in NLP
The natural language processing community has found that the choice of in-context examples has a huge impact on performance [1, 13J. Moreover, the way how in-context examples are constructed can also affect performance, e.g., prompt length and order of in-context examples. These findings prompted the NLP community to study how to find good in-context examples for large language models, which largely inspired our research. Liu al. [13] assumed that good in-context examples should be semantically close to query sentences, based on which they proposed to select nearest neighbors in the training set measured by sentence encoder like RoBERTa [14]. Rubin el al. [18] first used an unsupervised method to retrieve some candidates, among which top examples were chosen using a supervised prompt retriever to maximize downstream performance. Different from the prompt retrieval research in NLP, work focuses large vision models, which different from language models in many ways and require customized designs to deal with visual input.
5 Discussion and Conclusion
Our research presents a timely study on an emergent ability termed in-context learning for large vision models. We systematically investigate how the choice of in-context examples impacts down- stream performance, exposing a critical issue that different in-context examples could lead to drasti- cally different results. We then propose effective p retrieval framework for visual in-context learning, with two simple implementations provided: one based on unsupervised learning and the other based supervised learning. Our methods obtain significant improvements over random lection under various problem settings, demonstrating huge potential of prompt retrieval methods in black-box adaptation scenarios for vision foundation models.
Our research also unveils some intriguing phenomena. For instance, we show that a good in-context example should be semantically similar to the query and closer in context. A model that can better balance spatial and semantic closedness in feature space would be more ideal for visual in-context learning. We hope the insights presented in this work could pave the way for developing more effective prompt retrieval methods.
Our experiments show that our methods are not strong enough to cope with distribution shifts. Though our methods outperform random selection under distribution shifts, the than that on a standard benchmark, suggesting huge room for improvement.
gap is much smaller

---

Acknowledgement
This study supported by the Ministry of Education, Singapore, under its MOE AcRF Tier (MOE- '2EP20221-00 -0012), NTU NAP, and under the RIE2020 Industry Alignment Fund Industry Collab- oration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
10

---

30.33
IoU: 56.60
IoU: 47.70
IoU: 85.00
IoU: 25.80
IoU: 24.85
74-10
loO: bo.85
loU: 73.68 (c)
lol: 74-72
lol: 36.b5
(d)
(e)
1oU:31.63
IoU: 27-79
IoU: 12.32
37.01
loU: 48.08
IoU: (g)
IoU: 30.71 (h)
IoU: 8o.97 (i)
IoU: ş1.86 86 (
loU: 63.18 (k)
loU: 53-17 (l)
IoU: 70.98 (m)
IoU: 75.23 (n)
IoU: 65.87
IoU: 51.86 (p)
loU: 80.37 (r)
loU: 82.55
(s)
Figure 6: In-context examples, which are from the foreground segmentation task, retrieved by Un- supPR and SupPR. These grids show examples from the train, tv, and bus categories.
A Illustration of In-context Examples
In the supplementary material, illustrate in-context learning results of foreground segmen- tation, single object detection, and colorization tasks.
A.1 Foreground Segmentation
The main paper presents the in-context examples from the person and cow categories. In the supple- mentary, shown in Fig. 6-11, present examples from the remained 18 categories in Pascal-5'.

---

loU: 38.64
35.35
IoU: 16.38
40.54
19.90
IoU:
loU: 66.08 (a)
loU: 54-57 (b)
IoU:
lol: 66 (d)
loU: (e)
IoU: 30.44 (f)
(c)
74, 64
IoU: 43.23
78.78
34.12
43:23
IoU: 4.00
loU: 77.60 (g)
loU: 54.25 (h)
81.50 ())
65.49
66.75 (k)
IoU: 35.28 (l)
IoU: 5.50
loU: 38.79
27.84
67.02
IoU: 7-34
IoU: 29.58
IoU: 13.89
IoU: 47-04
52.09
73.39 (p)
33-79
IoU: 61.31
(o)
Figure 7: In-context examples, which are from the foreground segmentation task, retrieved by Un- supPR and SupPR. These grids show examples from the bottle, sheep, and bird categories.
A.2 Single Object Detection
As shown in Fig. 12-13, illustrate the in-contest examples from the single object detection task. By comparing the in-context examples picked by UnsupPR and those picked by SupPR, we find the examples found by SupPR more similar to the queries in terms of object pose (e.g., Fig. 12(l), viewpoint (e.g., Fig. 12(r).
A.3 Coloralization
shown in Fig. 14-15, we illustrate the in-context examples from the colorization task. This task aims to map gray-scale image to color image. By comparing the in-context examples picked
12

---

17:95
loU: 56.08
IoU: 57-14
IoU: 28.07
1o5:
IoU: 38.60
loU: 29-17 (a)
IoU: 71.44 (b)
loU: 70.87
3oU:34.96 (d)
14 14.59 (e)
74.274.21
loU: 43.06
IoU: 61.63
IoU: 52.62
IoU: 1.96
loU: 58.01
IoU: 62. 24
loU: 52.98 (g)
loU: 64.74 (h)
loU: 63.29 (i)
loU: 41.3a (j)
loU: 72.49 (k)
loU: 68.o4 ()
loU: 5.50
IoU: 8.06
IoU: 6.96
IoU: 6.69
IoU: 11.12
IoU: 5.24
iMi
IoU: 13.89 (m)
IoU: 26.55 (n)
IoU: 25.93 (o)
IoU: 21.72 (p)
loU: (r)
loU: 8.52
(s)
Figure 8: In-context examples, which are from the foreground segmentation task, retrieved by Un- supPR and SupPR. These grids show examples from the boat, airplane, and bicycle categories.
by UnsupPR and those picked by SupPR, we find the ground truth images of examples found by SupPR are more similar that of the queries in terms of image style, e.g. the background color (e.g., Fig. 14(g)
13

---

IoU:
67.96
IGU: 59-41
63.17
loU: 63-77
RETEO DITEE
PEPAD U
lou: 5.00 (a)
26.03 (b)
IoU: 84.00 (c)
loU: 76.11 (d)
loU:
loU: 75:39 (f)
(e)
IoU: 34.29
IoU: 28.87
IoU: 62.26
Iol: 33.00
IoU:
1oU:9.00
IoU: 51.92 (g)
loU: 48.62 (h)
72.57
loU: 51.00 (i)
loU: 57.00 (k)
loU: 92.00 (l)
5.49
10. 53
2B.56
0.25
23-46
IoU:
IoU: 23-59 (n)
IoU: 42.44
loU: 40.91 (p)
loU: 36.67
loU: 26.25 (6)
Figure 9: In-context examples, which are from the foreground segmentation task, retrieved by Un- supPR and SupPR. These grids show examples from the car, cat, and chair categories.
14

---

IoU: 67.09
IoU:39.74
IoU: 65.46
loU: 22.37
IoU: 22.38
IoU: 17.91
IOU: 71.61
loU: 59.27 (b)
150:85:94 (c)
lou:
1oU:7 (e)
IoU:
(a)
(d)
(f)
IoU: 43.22
IoU:
IoU: 60. o0
loU:
loU: 26,76
loU: 66
loU: 66.82 (g)
loU: 6g.63 (h)
lol: 7o.87
loU: 68.91
lol:: 72.77 (k)
loU: 72-97
IoU: 25.92
IoU: 0.00
loU: 21.99
IoU:
IoU: 73-70
loU:
IoU: 70.16 (m)
IoU: 68.04 (n)
loU: (o)
loU: 54.00 (p)
1o.7 (n
loU: 51.60 (s)
Figure 10: In-context examples, which are from the foreground segmentation task, retrieved by UnsupPR and SupPR. These grids show examples from the dog, horse, and motorbike categories.
15

---

MIN
IoU: 0.00
IoU: 56.38
IoU: 63.09
IoU: 0.00
loL: 28.82
IoU: 2.23
lou:
IoU:
IoU: 93.05 (c)
loU: 6:45 (d)
10U:45.35 (e)
IOU:
(a)
(b)
(f)
IoU: 43-73
IoU:
IoU: 27.15
IoU: 23.25
loU: 2.44
IoU: 29.16
IoU: 46.55 (g)
IoU: 18.00 (h)
IoU: 33-91
IoU: 65.69 (j)
loU: 33-75 (k)
loU: 35.21
IoU: 34.58
loU: 7.08
IoU: 0.00
IoU: 34.88
loU: 8.34
loU: 0.74
IoU: 60.62
IoU:
IoU: 27.09
IoU: 45.71 (p)
loU: 42.56
loU: 47.07
Figure 11: In-context examples, which are from the foreground segmentation task, retrieved by UnsupPR and SupPR. These grids show examples from the table, plant, and sofa categories.
16

---

lo 25.88
IoU: 26.82
loU: 33.28
loU: 10.59
lol: 33-40
loU: 44-57
loU: 51.28 (a)
IoU: 67.47 (b)
IoU: 59-97 (c)
loU: 40.88 (d)
lol: 66.10 (e)
loU: 68.02
IoU: 0.00
IoU:
IoU: 3.23
IoU:
IoU: 23.47
3o0:3 30.23 (g)
IoU: 22.71 (h)
IoU: 54.34
IoU: 32.34 (i)
lol: 27-73
0.00
loU: 21.28
IoU:1.73
IoU: 20.43
loU: 36,78
loU: 11.90
loU: 27.65
loU: 5a.64
loU: 61.60 (o)
IoU: 71.30 (p)
loU: 58.09
loU: 32.60
In-context examples, which are from the single object detection task, retrieved by Un- supPR and SupPR. We find the examples found by SupPR are more similar the queries in terms of object pose (e.g., (f)), viewpoint (e.g., (r))
Figure 12:
17

---

loU: 6.89
10U:15:34
loU: 32.98
IoU: 31.79
IoU: 14.15
loU: 39.50
IoU: 29.59 (a)
IoU: 73-74 (b)
IoU: 67:24 (c)
loU: 63.17 (d)
loU: 69.75 (c)
loU: 67.26 (f)
loU: 26.33
IoU: 36.21
3:23
loU: 13.24
loU: 24.76
loU: 53.40 (g)
lol:: 45.88 (h)
loU: 1oU:71.97 71-97 ())
IoU: 32:34
loU: 52.23
58.6
loU: 20.72
IoU: 0.00
IoU: 41.23
20.43
IoU: 8.05
loU: 4.51
IoU: 42.64 (m)
IoU: 29.62 (n)
loU: (o)
1o.3 (p)
loU: 45-97 (r)
1.26
(s)
Figure 13: In-context examples, which are from the single object detection task, retrieved by Un- supPR and SupPR. We find the examples found by SupPR are more similar the queries in terms of object pose (e.g., (I)), viewpoint (e.g., (m))
18

---

:3.33
0.79
0.85
mse: 0.66 (a)
mse 0.53 (b)
mse
mse 0.49 (d)
mse
mse 0.78
(c)
(e)
(f)
2.16
mse m.58
1.15
mse: 0.99
mse: 0.73 (g)
msee 0.40 (h)
mse 0.23 (i)
msse 0.88 ())
mse: 0.17 (k)
mse 0.48
Figure 14: In-context examples, which are from the colorization task, retrieved by UnsupPR and SupPR. We also show the ground truth the query image. The query image is the gray-scale version of its ground truth. The ground truth images of the in-context examples found by SupPR more similar than those found by UnsupPR to the ground truth images queries in terms image style, egg, the background color (g).
19

---

adidas
Tasio
mse: 1.01
mse 51
msee 1.25
mse: 0.66
mse 0.79
mse 0.78
mse: 0.64 (a)
0.80 (b)
mse (d)
mse: (e)
mse 0.55 (f)
(c)
0.79
11.19
:3:48
0.99
0.80
0.52 (g)
mse: (h)
mse:
mse: 0036
mse: 00.3 (k)
())
Figure 15: In-context examples, which are from the colorization task, retrieved by UnsupPR and SupPR. We also show the ground truth of the query image. The query image is the gray-scale version of its ground truth. The ground truth images of the in-context examples found by SupPR are more similar than those found UnsupPR to the ground truth images of queries in terms of image style, e.g. the background color (h).
20

---

References
[1] Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke 7ettlemoyer, and Marjan Chazvininejad. In-context examples selection for machine translation. arXiv preprint arXiv:2212.024477, 2022.
[2] Jean-Baptiste Alayrac, Jeff Donahue. Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican. Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
[3] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Vi- sual prompting via image inpainting. Advances in Neural Information Processing Systems (NeurIPS), 35:25005-2500 2022. 2,7,,
[4] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 33:1877-1901, 2020.
2020. 3,9
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 11929, 2020. 6
[7] Mark Everingham, SM Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and An- drew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision (ICCV), 111(1):98-136, 2015.
[8] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learn- ing at scale. arXiv preprint arXiv:2211.07666 2022.
[9] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoeneoders scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, 2022.
[10] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuming. arXiv preprint S1t.v::2004006991 2021. 1
[11] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 101.00090, 2021.
[12] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision (ECCV), pages 740-755. Springer, 2014.
[13] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.0688 2021. 9
[14] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1669, 2019.
1692,
[15] Junting Pan. Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: earfmeter-efffciient image-to-vides transfer learning for action recognition. arXiv preprint arXiv:2206.1355 2022.
[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages
PMLR, 2021. 2,
21

---

[17] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and lya Sutskever. Zero-shot text-to-image generation. International Conference on Machine Learning (ICML., pages
PMLR, 2021.
[18] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633, 2021.
[19] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 115(3):211 1-252, 2015. 5
[20] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint a7Xiv 2022. 1
[21] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. british machine vision conference (BMVC), 2017. 2,5
[22] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499, 2022.
[23] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.0467. 046776 2022.
[24] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Trunsactions Patlern Analysis and Machine Intelligence (TPAMI), 2022. 6
[25] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learn- ing for vision-language models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[26] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), 2022.
22


Reviews:
[{"review_id": "uUUZ9k0Tzg", "reviewer": "Reviewer_mHCs", "summary": "This paper aims to address the problem of vision in-context-learning that the performance highly depends on the choice of visual in-context examples. In the paper, the authors propose automatically retrieving prompts in unsupervised and supervised ways without reaching internal weights of large vision models. Besides, this paper comprehensively studies how to select good examples for visual in-context learning and shares some valuable insights with the community on choosing good visual in-context examples.", "questions": "1. The paper discusses the impact of the order and number of in-context examples. While more in-context examples generally lead to better performance, is there an upper limit to this? Will adding more examples become redundant or even detrimental at some points?\n\n2. The paper compares the proposed methods with random selection. Are there other baselines that the authors considered for comparison?\n\n", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper discusses several aspects that influence visual in-context learning and propose methods to choose in-context learning samples automatically to optimize visual in-context learning with inpainting method. The proposed methods both outperform random selection baseline and the supervised version requires training performs better than the unsupervised method in several tasks. Also, this paper discusses other factors that might influence visual in-context-learning including the number of examples, order of examples, and size of retrieval set, which shares some useful practical experience to the community.", "weaknesses": "1. This paper only discusses the previous paper of using inpainting as visual in-context learning as the visual icl frame and experiment on the dataset from that paper. It's hard to tell if experiments and conclusions conducted on a specific framework will generalize to more general \"visual in-context learning\" settings. Besides, although the supervised example retrieval method outperforms the unsupervised one, the additional model and training seem to be contradictory to the main advantages of ICL learning that requires no additional training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and comments. We respond to the concerns below:\n\n**Q1**:Whether this method can be generalized to more general \"visual in-context learning\" settings.\n\n**A1**:Thanks for your suggestion. Please refer to the **General Response** for the detailed explanation.\n\n**Q2**: SupPR contradicts the main advantages of ICL learning that requires no additional training\n\n**A2**: The advantages of In-context learning (ICL) comes from no additional training for the **pre-trained model** that might be only provided as the service, e.g. ChatGPT, Flamingo, PaLM2. SupPR only uses the pre-trained model as a mechanism for in-context results output, while training an independent scoring model for prompt retrieval. This approach uses downstream few-shot data effectively and do not train the pre-trained model. The goal of SupPR  is to enhance the ICL capability (such as foreground segmentation) emerged during pre-training rather than creating new capabilities.\n\n**Q3**: The upper bound of the number of in-context examples.\n\n**A3**: We appreciate the reviewer's suggestion to investigate the upper bound of the number of in-context examples. In response, we conducted an additional experiment focusing on the foreground segmentation task, where we employed SupPR as the prompt retrieval method. We increased the maximum number of in-context examples to 32. However, we would like to note that any further increase in in-context examples would result in out-of-memory errors on NVIDIA V100 GPUs. The findings, presented below, detail the performance improvement with an increase in the number of shots. The performance enhancement achieved from the latest shot is indicated within round brackets. From this table, it is evident that while there is an almost linear increase in performance from 1 shot to 16 shots, the improvement diminishes from 16 shots to 32 shots. \n\n|            | 1 shot | 4 shot        | 8 shot       | 16 shot      | 32 shot      |\n|------------|--------|---------------|--------------|--------------|--------------|\n| Seg.(mIoU) | 21.90  | 25.04 (+3.1) | 28.45 (3.4) | 31.25 (+2.8) | 32.64 (+1.4) |\n\n**Q4**: Other baselines that the authors considered for comparison\n\n**A4**: Thank you for your query on baselines. In line with Bar et al.[1], our primary baseline employs a within-class random selection of prompts matching the query image class. To broaden our comparisons, we introduced another baseline where prompts are randomly chosen from the entire training dataset. The significant performance difference between UnsupPR/SupPR and this (entire) Random baseline underscores the effectiveness of our method in real-world scenarios. \n\n|            | (entire) Random | (within-class) Random | UnsupPR | SupPR |\n|------------|-----------------|-----------------------|---------|-------|\n| Seg.(mIoU) | 24.46           | 27.45                 | 33.56   | 35.56 |\n\n[1] Bar, Amir, et al. \"Visual prompting via image inpainting.\" Advances in Neural Information Processing Systems 35 (2022): 25005-25017.\n\n"}, {"review_id": "2vPt7Ut3Pt", "reviewer": "Reviewer_QAph", "summary": "This paper investigates in-context learning for large vision models. Specifically, the authors propose to automatically retrieve prompt for vision models in two methods: (1) nearest example search with off-the-shelf model, (2) supervised prompt retrieval method.\nExperimental results show that the proposed method bring non-trivial improvement, comparing to the random selection on foreground segmentation, single object detection and colorization. ", "questions": "please refer to the weakness section", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n- the motivation of this paper is important: investigating how vision models could benefit from in-context learning.\n\n- the writing is clear and easy to follow.\n\n- The experiments show improvement comparing with random selection.\n\n", "weaknesses": "1. Visual in-context learning emerged from large autoregressive language models. In-context learning was referred to use language models for a wide range of downstream tasks without updating the model itself. But this paper, though claims to study visual in-context learning, only studies a limited set of tasks (assign labels to pixels) with in-paining model. I wonder if the authors could provide thoughts on how to apply to a wider visual tasks, like classification, or standard segmentation tasks (semantic and instance).\n\n\n2. The finding of \"a good in-context example should be semantically similar to query and closer in context\" seems to be a problem if the testing examples are unique, and there is no semantically similar {image, annotation} pair in the retrieval pool. This goes to the practicality of the method, and goes to the claim \"potential of using prompt retrieval in vision applications\".\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and comments. We respond to the concerns below:\n\n**Q1**:Only studies a limited set of tasks (assign labels to pixels) with an inpaining model; to a wider visual tasks, like classification, or standard segmentation tasks (semantic and instance).\n\n**A1**: Thanks for your suggestion. We would like to clarify that the SupPR/UnsupPR model primarily aims to enhance the in-context learning capabilities emerged during pre-training, rather than creating new abilities. Consequently, we posit that our model could be feasibly extended to standard segmentation tasks, provided the pre-trained inpainting model demonstrates in-context learning on such tasks, albeit its current limitation in standard segmentation tasks. \n\nAdditionally, our method shows potential for extension to a broader range of visual tasks. The UnsupPR/SupPR can be easily extended to other tasks. As detailed in the **General Response** section, we extend our method to image caption task and VQA task based on a vision-language model. We appreciate your feedback and encourage you to refer to that section for a comprehensive understanding.\n\n**Q2**: Seems to be a problem if the testing examples are unique\n\n**A2**: We appreciate the reviewer's point regarding unique testing examples. However, it is essential to note that our method is fundamentally designed for scenarios where there are a certain amount of testing examples, which aligns with the standard presumption of in-context learning. Also, due to the relative nature of similarity, the most semantically similar pair can always be identified within the retrieval pool. Our Figure 5 (left) illustrates that even with a retrieval set of just 0.01% of the entire dataset (about 30 image-annotation pairs), which may not include the most semantically similar pairs, supPR and UnsupPR consistently surpass random selection. This highlights the resilience and practicality of our method.\n"}, {"review_id": "DomHs3uHnx", "reviewer": "Reviewer_EQqT", "summary": "The paper identifies that visual prompting is sensitive to the choice of input-output example(s). To address this, the authors propose a retrieval framework to better select the examples. The authors propose supervised and unsupervised retrieval approaches which significantly improve the performance compared to random selection.\n", "questions": "1. The main weakness is the assumption that we have a set of tagged examples. In this case, why should we retrieve rather than use our tagged examples? e.g., train a new model overall tagged examples or use an ensemble? Perhaps the best way to address this would be to add another ensemble/supervised-trained baseline in Figure 5 (left). These baselines should utilize the same pool of examples.\n\n2. The supervised/unsupervised similarity function is straightforward but currently very specific to mIOU similarity. Can we improve this by utilizing similarity in feature space? (e.g., using existing CLIP or DINO, etc?). To be fair, from the empirical results it seems like by optimizing mIOU there is somewhat improvement for colorization and single object detection as well.\n\nMinor:\n\n3. I think there is a broader spectrum that can be analyzed for visual prompts. E.g., using synthetic task data as examples, using examples from different classes, then examples from within classes as analyzed in the paper.", "limitations": "The authors clearly stated and discussed the limitations of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Through extensive empirical study, the authors demonstrate the role of input-output examples for visual prompting. \n- The authors present two different retrieval strategies to choose the best visual prompting example and find that both approaches are superior to random choice.\n- The authors show that choosing the right example can also improve performance under distribution shifts. \n- It is also nice that the retrieval similarity function is class-agnostic. But regardless, empirically it extends to three different tasks. ", "weaknesses": "Overall, the main two weaknesses in my opinion are:\n1. The assumption that we have a set of tagged examples to retrieve from.\n2. The supervised/unsupervised similarity function is straightforward but currently very specific to mIOU similarity.\n\nMinor:\n\n3. I think there is a broader spectrum that can be analyzed for visual prompts.\n\nPlease see the questions for extended comments/feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and comments. We respond to the concerns below:\n\n\n**Q1**: The assumption that we have a set of tagged examples to retrieve from. Why should we retrieve rather than training our tagged examples?\n\n**A1**: It is essential to note that our method is fundamentally designed for scenarios where there are a certain amount of tagged examples, which aligns with the standard presumption of in-context learning. \n\nIn addition, we conduct studies to demonstrate the advantages of prompt retrieval. The following table, referring to Figure 5 (left), compares full-set and 1% of the full-set scenarios. The first two columns compare the performance of our SupPR model with the fine-tuned MAE model (the supervised-trained baseline that might concern you). Importantly, SupPR outperforms the fine-tuned model, especially when training data is scarce, underlining the superiority of the prompt retrieval approach. \n\nFurthermore, to accentuate the potential of visual in-context learning via prompt retrieval, we present an upper-bound performance, achieved by manually selecting the best in-context samples from the training dataset for each query. The noticeable performance difference between this upper bound and the fine-tuned MAE model highlights the potential and efficacy of prompt retrieval. If optimized, it can substantially enhance the performance of visual in-context learning, spurring further research in this field.\n|            | Fine-tune (MAE) | SupPR | Upper bound |\n|------------|-----------------|-------|-------------|\n| Seg.(mIoU) 1% full-set| 20.15           | 30.22 | 34.53       |\n| Seg.(mIoU) full-set| 34.98           | 35.56 | 40.23       |\n\n\n\n**Q2**: Supervised/unsupervised similarity function is straightforward but currently very specific to mIOU\n\n**A2**: Our similarity function indeed uses cosine similarity between two CLIP embeddings in the feature space. And, it's might be a misconception that it's inherently specific to mIOU. We emphasize that our methods have applicability beyond mIOU, including diverse generative tasks like image caption generation and VQA. Please refer to the **General Response** section, demonstrating the extendability of our method to other vision in-context learning tasks (image captioning and VQA), not limited to inpainting.\n\n**Q3**: A broader spectrum that can be analyzed , e,g. using examples from different classes, then examples from within classes\n\n**A3**: Thank you for the insightful suggestion to broaden the analysis spectrum. In response, based on foreground segmentation task, we conducted random sampling experiments with three distinct scenarios: \n\n(a) examples randomly sampled from the entire training dataset\n\n(b) examples randomly sampled across different classes\n\n(c) examples sampled from within classes.\n\nOur findings indicates that the alignment of example data within the same classes as the query data is crucial. This reaffirms our observation that an effective in-context example should be semantically similar to the query.\n\n|            | Different classes | Whole training data | Within classes | UnsupPR | SupPR\n|------------|-------------------|---------------------|----------------|-------|------|\n| Seg.(mIoU) | 24.02             | 24.46               | 27.45          | 33.56 | 35.56\n\n"}, {"review_id": "fKtBwTqvwE", "reviewer": "Reviewer_vwKB", "summary": "This paper studied in-context learning abilities of large vision models and finds downstream task performance to be highly sensitive to the choice of examples. They observe that the closer the in-context example is to the query, the better the performance. Since manually designing prompts would be time-intensive, they propose a (contrastive) supervised and an unsupervised version of a framework for prompt retrieval guided by a score function. They evaluate the proposed framework on the foreground segmentation, single object detection and image colorisation tasks.", "questions": "1. It is surprising to me that the smallest gains are observed on the colorisation task and supervision is not helpful, despite the fact that both training and evaluation are done on ImageNet. Do you have any intuition for why this may be the case?\n2. In Sec 3.3, you mention that 20% of the data is enough for good performance. Is this 20% of 50,000, i.e. 1000 images?\n3. What is the intuition behind MAE in Table 5 performing worse than UnsupPR?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- In-context learning in vision is a new and active area of research and methods for selecting better examples for in-context learning will likely be of interest to the computer vision community. \n- The proposed method for prompt retrieval outperforms randomly selected examples on all tasks considered.\n- Distribution shift results are interesting, suggesting that the supervised prompt retrieval method acquires in-context knowledge that is robust to distribution shift.\n- The analysis and ablations are interesting and insightful.", "weaknesses": "- The finding that examples in the prompt should be semantically close to the test example is known from language and not that surprising.\n- The proposed method is technically relatively straightforward compared to an average NeurIPS paper.\n- The analysis is limited to one model for in-context learning and may not transfer to other models.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and comments. We respond to the concerns below:\n\n**Q1**: Semantically close to the test example is known from language and not that surprising.\n\n**A1**: Yes, we acknowledge the parallels between some of our observations and those from the natural language domain. However, most of our study presents findings that are significantly novel to the computer vision community. For example, while the concept of semantic similarity might seem intuitive initially, the specific attributes that constitute this similarity in the context of computer vision remain largely unexplored. As highlighted in Figure 3, elements such as background, pose, appearance, and viewpoint play crucial roles. \n\nFurthermore, our study identified differences between language and vision models. Notably, while in-context learning in language models shows sensitivity to the sequencing of in-context examples[1], such order doesn't significantly influence vision models. This underscores the unique challenges and considerations intrinsic to computer vision tasks.\n\n**Q2**: Technically relatively straightforward compared to an average NeurIPS paper.\n\n**A2**: Our proposed methodology, specifically for the SupPR, introduces an innovative approach to prompt retrieval that efficiently boosts the downstream capabilities of the pre-trained model without fine-tuning it. In addition, as shown in table.5, the superiority of prompt retrieval becomes apparent only when we meticulously develop an appropriate learning pipeline, such as our SupPR, which effectively leverages this feedback to enhance the downstream performance of the pre-trained model. We believe our prompt retrieval design is a beneficial contribution to the community.\n \n**Q3**: One model for in-context learning and may not transfer to other models\n\n**A3**: Thanks for your suggestion. Please refer to the **General Response** for the detailed explanation.\n\n**Q4**: Smallest gains are observed on the colorization task and supervision is not helpful\n\n**A4**: A possible explanation for this observation is that the current pre-trained inpainting model does not efficiently perform in-context learning on the colorization task (for a detailed illustration, please refer to Figures 14-15 in the supplementary material). If the pre-trained model's in-context learning ability for the colorization task is weak, we face difficulties in ranking positive and negative examples relative to a query image. This, in turn, impedes the effective training of a robust feature extractor for SupPR.\n\n**Q5**: 20% of the data is enough for good performance\n\n**A5**: Given that each data split comprises approximately 3,000 samples, this implies that a subset of 600 samples (20% of the data) is deemed sufficient for effective model performance.\n\n**Q6**: Intuition behind MAE in Table 5 performing worse than UnsupPR\n\n**A6**: The differential in performance is fundamentally attributed to the complexity of designing a robust feature extractor for SupPR. The strength of SupPR comes from our intellectual design rather than the supervision signal. Without such careful design, the SupPR can even perform worse than the UnsupPR.\n\nMoreover, this finding highlight that semantic similarity is a key factor in selecting an effective prompt. The inferiority performance of the fine-tuned MAE suggests that features optimized by foreground segmentation may be deficient in encoding semantic information. On the other hand, given CLIP's well-known ability in encoding semantic information, our UnsupPR is equally good at retrieving semantically similar examples.\n\n[1] Agrawal, Sweta, et al. \"In-context examples selection for machine translation.\" arXiv preprint arXiv:2212.02437 (2022)."}]

Author rebuttal:
**General Response**

Dear Reviewers, 

We sincerely appreciate the time and insightful comments provided by all reviewers, which have been instrumental in enhancing our paper. We are encouraged by the positive feedback regarding the motivation behind our work (QAph) and our strong performance across various tasks (mHCs, QAph, EQqT, vwKB), our analysis is useful, interesting and insightful (mHCs, vwKB). 

**We would like to highlight the key contributions of our work**

(1) Our study is the first comprehensive examination of selecting effective examples for the emerging field of visual in-context learning. We uncover a critical issue: the choice of in-context examples significantly impacts performance.

(2) From a technical standpoint, we introduce a prompt retrieval framework that automates the prompt selection process, offering two straightforward implementations: an unsupervised method and a supervised method.

(3) Through extensive experimentation on three visual in-context learning tasks not encountered during pre-training (foreground segmentation, single object detection, and image colorization), we provide the community with valuable insights on identifying suitable visual in-context examples. For instance, our findings indicate that the supervised method consistently outperforms other approaches and often identifies examples that are both semantically related and/or spatially similar to a given query.

**Our methods are adaptable to broader "Visual In-Context Learning" scenarios**

Our strategies - SupPR and UnsupPR - are designed with a broad scope of applicability , capable of extending to a variety of "visual in-context learning" frameworks. We have substantiated this claim by conducting experiments on the COCO Caption dataset and the TextVQA dataset, utilizing the recently public OpenFlamingoV2-9B model [1]. OpenFlamingo, a visual-language model distinct from the inpainting model, can generate natural language description from visual-language inputs.

In these experiments:

- For the UnsupPR method, we used the CLIP ViT L-14 as the feature extractor.
- For SupPR, a prompt is defined either by an image-caption pair (COCO caption) or by image-question-answer triplets (TextVQA). A prompt that instructs a query image for the maximum/minimum CIDEr (COCO caption) or accuracy (TextVQA) is labeled as positive/negative prompt. Using these positive/negative prompts, we trained a new feature extractor following the methodologies outlined in Sec.2.2.2.

The comparative performance between random selection and our proposed SupPR and UnsupPR are presented below. The results indicates the broader applicability of our methods to more visual in-context learning task and visual model.

|                      | Random | UnsupPR | SupPR |
|----------------------|--------|---------|-------|
| COCO caption (CIDEr) | 84.2   | 87.3    | 88.9  |
| TextVQA (Acc.) | 25.4 | 30.2 | 33.4 |

Once again, we thank the reviewers for their invaluable feedback and support.

[1] Awadalla, Anas et al. “OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models.” (2023).


Meta-review: Thank you for addressing the concerns highlighted during the review phase in detail.

From your response, it's clear that you've actively engaged with the feedback, particularly regarding the method's extendability. The majority of reviewers are content with your clarifications and recognize the novelty, robustness, and relevance of your method within the "Visual In-Context Learning" context. Your focus on the importance of in-context example selection offers pivotal guidance for subsequent research in this area.

We're pleased to accept your paper, confident in its potential to significantly influence the visual in-context learning domain.

Area Chair, NeurIPS.
Decision: Accept (poster)