Towards Global Optimal Visual In-Context Learning Prompt Selection
Authors: Chengming Xu
Chen Liu
Yikai Wang
Yuan Yao
Yanwei Fu
Venue: NeurIPS 2024 poster (2024)
Keywords: in-context learning
Primary area: machine_vision

Abstract: Visual In-Context Learning (VICL) is a prevailing way to transfer visual foundation models to new tasks by leveraging contextual information contained in in-context examples to enhance learning and prediction of query sample. The fundamental problem in VICL is how to select the best prompt to activate its power as much as possible, which is equivalent to the ranking problem to test the in-context behavior of each candidate in the alternative set and select the best one. To utilize more appropriate ranking metric and leverage more comprehensive information among the alternative set, we propose a novel in-context example selection framework to approximately identify the global optimal prompt, i.e. choosing the best performing in-context examples from all alternatives for each query sample. Our method, dubbed Partial2Global, adopts a transformer-based list-wise ranker to provide a more comprehensive comparison within several alternatives, and a consistency-aware ranking aggregator to generate globally consistent ranking. The effectiveness of Partial2Global is validated through experiments on foreground segmentation, single object detection and image colorization, demonstrating that Partial2Global selects consistently better in-context examples compared with other methods, and thus establish the new state-of-the-arts.

Towards Global Optimal Visual In-Context Learning Prompt Selection
Chengming Xul Chen Liu²"
Yikai Wang
Yuan Yao? Yanwei Fu'
FFudan University
"Hong Kong University of Science and Technology {cmxu18, yanweifu)@fdu. edu. en cliudh@connect. ust hk yi-kai. wwwg@outlook.coomm com
yuanySust hk
Abstract
Visual In-Context Learning (VICL) is prevailing way to transfer visual foun- dation models to new tasks by leveraging contextual information contained in in-context examples to enhance learning and prediction of query samples. The fundamental problem in VICL is how to select the best prompt to activate its power much possible, which is equivalent to the ranking problem of testing the in-context behavior each candidate in the alternative set and selecting the best one. To utilize more appropriate ranking metric and more comprehensive infor- mation among the alternative set, we propose a novel in-context example selection framework to approximately identify the global optimal prompt, i.c. choosing the best performing in-context examples from all alternatives for each query sam- ple. Our method, dubbed Partial2Global, adopts ranker to provide a more comprehensive comparison within several alternatives and consistency-aware ranking aggregator to generate globally consistent rank- ing. The effectiveness of Partial2Global is validated through experiments on fore- ground segmentation, single object detection and image colorization, demonstrating that Partial2Global selects consistently better in-context examples compared with other methods, and thus establishes the new state-of-the-an Code is available atttp::/githuu: com chtp///////a/////////ccc//actttt www.w/hwnkiineecccccccccc ranking in__ontext.gin
seaafoomer-basee list-wise
Introduction
Foundation models like those used in AIGC, such as GPT and Gemini, play big role in its success. Different ways to fine-tune these models are being suggested to use them more effectively. One popular idea is called Visual In-Context Learning (VICL), which is a way to teach Visual Foundation Models (VFMs) using context from images. This helps the models learn better and make more accurate 'guesses' based on what they see. Basically, VICL gives the models prompt examples to learn from, like pairs of images and labels, termed in-context examples, based on which query samples are inferred. This mimics how humans learn with the guidance. Using this method, VFMs can take lots of different tasks, like understanding point clouds (6), editing images 29] as well as multi-modal inference [34], even if they were not trained specifically for those tasks.
The main challenge of VICL picking the best in-context example that matches the query images, so the VFM can perform well. Empirically, we have found that randomly picking an example does not always work well, while carefully selected example can really boost performance, as observed in
These authors contributed equally to this work. "Corresponding author.
Dr. Fu is with School of Data Science in Fudan, Fudan ISTBI-ZINU Algorithm Centre for Brain-inspired Intelligence, Shanghai Key Lab of Intelligent Information Processing, and Technology Innovation Center of Calligraphy and Painting Digital Generation, Ministry of Culture and Tourism, China. Dr. Xu is now with Tencent Youtulab.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SupPR
6.11
3.51
Ours
28.10
60.92
67.77
72.37
72.43
91.05
57.92
60.42
Figure I: Qualitative comparison between our method and VPR, specifically SupPR, in foreground segmentation. In each item we present the image grid in the same order as the input MAE- VQGAN, ie. in-context example and its label in the first row, query image and its prediction in the second row. The loU is listed below each image grid.
previous works 32 21. and visualized comparison in Fig. Typically, VICL has set of options to choose from (like the training data). We can think of picking the best in-context example as a ranking problem want to choose the one that helps the most. Idcally, with a new image, rank how well they work, and then pick the best one. However, doing that for every image is not practical. So, one of the biggest challenges in VICL is figuring out how to do this ranking without actually testing each option.
would like to test each option
There are a few key challenges in choosing the best prompt for VICL. (1) Choosing the Right Metric: Since we cannot directly test how well a prompt works on specific image, we need to find a different way to measure it. Previous methods like VPR (32 mainly use two metrics from ranked observations the training set: visual similarity and pair-wise relevance instantiated through contrastive learning. However, visual similarity does not always work well as in our experiments. The contrastive learning method naturally restricts how much the model can learn because of how its objective function is designed. To make contrastive learning effective, VPR needs to collect the most representative positive and negative examples for each query, which are the best and worst in-context examples. Consequently, large amount of ranking observations remain unused during training. (2) Deciding on the Comparison Ser: When considering the ranking metric, it is best to exhaustively compare each example against all the candidates available. However, directly ranking all alternatives globally is often not always possible. Instead, we have to aggregate rankings from partial predictions. Pair-wise ranking. like in VPR, ensures consistent ranking predictions by comparing each option's similarity to the query individually. But it cannot grasp the overall relationship between options. Conversely, methods with larger capacity, such as list-wise ranking, struggle create a unified ranking due to inconsistencies among different partial predictions. Anyway, balancing feasibility and capacity is crucial for VICL sample selection algorithms.
To solve these problems, this paper proposes systematic VICL example selection framework, dubbed Partial2Global,, towards the global optimal in-context examples selection. Partial2Global involves several list-wise partial rankers based on transformers to rank multiple alternative samples, and a consistency-aware ranking aggregator achieve globally consistent ranking from the partial rankers. (1) To approximate the ground-truth ranking metric, we propose transformer-based list-wise ranker rank multiple alternative samples in takes features of several alternatives together with the query sample from pretrained foundation models as input. The ranking predictions can then be inferred with more comprchensive knowledge from multiple alternatives and query sample by merging information among class tokens regarding each sample. Through meta-learning based training strategy, our model can learn a generalizable ranking metric for each specific task, thus better facilitating VICL example selection. (2) To approximate the global ranking, we propose a consistency-awart aware ranking aggregator strike the balance between feasibility and ranking capacity. Particularly, partial and noisy ranking predictions from the trained ranking model are collected and divided into several groups, which can reorganized and aggregated by learning global ranker that has minimized average distance with all ranking groups. The resulted global ranking enjoys better ranking consistency and thus provides better in-context examples to boost VICL performance.
informative and comprehensive manner. This model

---

sorted alternatives
partial
query
query
sorting
best
0.3
sort
worst
eiternatvee
pair ranker
0.4
list ranker
ranker
best
0.9
worst
(a)
(b) list-wise ranker nalve aggregation
alternatives
query
g global ranking
best
shuffle-
best
list-wise renker
eggregstor
worst
observation pool
shuffled veriants
[e] üss-wise ranker with consistency-awarc aggregator
Figure 2: Systematic comparison between three different frameworks for in-context example selection. (a) VPR, which uses pair-wise ranker trained with contrastive learning to calculate the relevance score for each alternative. (b) List-wise ranker with naive aggregation, in which alternatives are split into non-overlapped subsets. These subsets are ranked with the proposed list-wise ranker. Then we iteratively select the best example in each subset and rank them. (c) List-wis wise ranker with consistency-awan aggregator, in which alternatives are first shuffled and predicted with list-wise ranker into an observation pool. These partial observations are then aggregated with the proposed aggregator to achieve global ranking.
In order validate the effectiveness of our method, we conduct experiments on three different tasks including foreground segmentation, single object detection and image colorization with datastet including Pascal VOC and ImageNet, following the previous works. Extensive results show that Partia/2Global not only can provide consistently better in-context examples, which results in new state-of-the-art performance, but also shed light on selection heuristics in-context examples, facilitating further research on visual in-context learning.
Contributions of this work
follows:
We emphasize the importance the global consistent ranking relationships and correct ranking
metric approximation in the selection of in-context samples for VICL.
We propose to approximate the optimal global ranking with a transformer based list-wise ranker, which can process
comprehensive information and produce stronger predictions.
We adopt a consistency-awant ranking aggregator to aggregate the partial predictions generated by ranking model.
Extensive experiment results show that our method consistently works well in various visual tasks
based on pretrained in-context learning model, showing the practical value of such method.
2 Related works
In-context learning, Given the current trend of scaling up model sizes, the large-scale models such large language models (LLMs) 2 and their multi-modal counterparts [10] shown gradually learn the ability to perform in in-context learning, i.e. inference with knowledge provided by few-shot context samples, rather than training the model with extra data. For example, Pan al. [IS] designed an in-context learning system to generate symbolic representation, which can then be used for logical inference. Zhang et. al. [33 proposed to leverage in-context learning to edit factual knowledge in LLMs. Handel et. al. 7 showed that in-context learning works by compressing training set into task representations which can modulate models during inference for desired output. Such property enables nonprofessional users to drive the model simply with several examples, the same as

---

someone teachers his children. Recently, MAE-VOGAN [ and Painter (24] show that by learning to inpaint the image grid composited by support and query pairs, the vision models can also learn the in-context learning ability. VPR [32 follows MAE- VQGAN and focuses on the specific problem of selecting good in-context examples. They propose a simple strategy by learning a contrastive learning based metric network with performance metrics. Prompt-SelF [21] analyzes VPR's s strategy and extends it to take into account both pixel-level visual similarity and patch-level similarity. Moreover, prompt-SelF proposes an test-time ensemble strategy gather predictions with different permutation of in-context image grids. Our work inherits the idea of VPR to concentrate on in-context example selection. Different from VPR, stress the importance of make full use of training sample, and the post-process of model predictions. To this end, we propose a novel framework which first trains ranking models with the performance metric data and then process the ranking predictions with consistency-aware ranking aggregator.
Sample selection and Ranking in deep learning. Sample selection is crucial for improving model training [20]
13 27 26 25 In in-context learning, sample selection focuses on finding the best prompt example for given query. This is typically done by training additional ranking models. These models compare a list of examples with the query based on their similarity in a specific property. Burges et al. 3 proposed an objective function for pairwise ranking, which was further extended to listwise ranking [23]. Recently several works focus on learning ranking with transformer structures. For example, Kumar et al. [9] proposed to adopt attention mechanism to interact between samples to be ranked. Liu et al. [I]] proposed to build the ranking model based on GPT model. In this paper we propose to leverage the learning rank technique to build transformer- based list-wise ranker, which can learn more comprehensive information than previously used pair-wise ranker learned with contrastive learning.
Aggregating partial ranking predictions. Jiang et. al. 8 proposed HodgeRank as a Hodge- theoretic approach to statistical ranking, which can provide global ranking with incomplete and inconsistent ranking data. The comprehensive insight of HodgeRank made it widely applicable to quality of experience assessment. For example, Xu et al. (30) adopts HodgeRank for subjective video quality assessment based on random graph models. The similar idea is also applied to crowdsourced pairwise ranking aggregation [3]] In this paper we take inspiration from Hodge theory to build novel framework for visual in-context learning. While such a strategy has never been studied before, we show that our framework can significantly boost the visual in-context performance by selecting better examples.
3 Methodology
Visual in-context learning, Visual In-Context Learning (VICL) aims to build the inference procedure of each testing sample based on knowledge provided by in-context examples. MAE-VQGAN [I] instantiates such an idea through image inpainting. Concretely, given query sample, an example image selected randomly along with its annotations (e.g.. bounding boxes, masks, etc.) from training set. Then the example image, example label, query image and a random query label be predicted are arranged as an image grid so that the query label is placed in the lower right corner. Then this composed image is used as input to an inpainting model. The masked patches are predicted by a transformer autoencoder and further decoded with the pretrained VQ-VAE [22. To ensure the generalization ability. MAE-VQGAN is pretrained on large-scale dataset collected from arxiv papers. While it shows promising results among different kinds of tasks, the important problem of in-context example selection is omitted, which leaves this method great potential for improvement.
Preliminary:
Preliminary: Visual Prompt Retrieval Visual Prompt Retrieval (VPR) (2) mainly focuses on the in-context example selection based on MAE- VQGAN. Specifically, VPR starts with the heuristic that more similar an image to the query image, the better it is as the in-context example. Based on this, the unsupervised and supervised variants of VPR are designed. In the unsupervised setting, VPR directly selects the most visually similar training images for each query image. As for the supervised setting, VPR first creates of a performance metric dataset collected from the training set, symbolized as Xx yf Nr. where 2g denotes query sample, Xa denotes a set of K alternatives to be ranked, and y denotes the performance metric, e.g. IoU for segmentation or accuracy for classification, for pretrained MAE-VQGAN when testing T. on specific task using X'x as in-context examples respectively. Then a metric learning model is trained to learn the in-context performance of training samples, i.c. the best performing samples are labeled as positive samples and learned

---

against the worst performing ones as negative samples. Such a method, while outperforming the naive MAE-VQ -AANV still cannot fully explore the knowledge regarding the ranking of in-context examples with the unprocessed partial observations. To this end, we propose a novel method to boost VICI. with more proper in-context example selection.
Overview. To boost VICL with better in-context example selection, we in this paper propose a novel pipeline including two main steps. First we train a ranking model (Sec. 3.1) based on the performance metric dataset collected in the same way as VPR. Different from VPR. we utilize an extra transformer model learn list level ranking instead of instance level scoring so that inner relationship between alternatives can be better utilized for ranking. Once the ranking model is trained, the global ranking of alternative set needs be gathered from partial ranking predictions. To this end, we propose to adopt a consistency-award ranking aggregator (Sec. (.2) to process predictions to provide globally consistent ranking results, thus better facilitating VICI..
3.1 Ranking in-context examples with list-wise ranker
In order to learn a more proper ranking metric approximation, we propose to adopt a transformer- based list-wise ranker. Our methodology is rooted in the principles of VPR. Given the query sample 2y and the alternative set Xg, we first sample a k size subset xr from Xg. Then a ranking model Φb is built to provide ranking prediction of as Φx (2, Concretely, features 2y e R(N+1)xC ,2 R (N+ corresponding to Z... 2 are extracted with pretrained transformer model @ such as CLIP [ or DINO E where N + features include class token and patch tokens, C' denotes feature channels. Following the feature extraction, we concatenate all these features to form feature sequence
[R(+I(N++) For the sake of simplicity and to facilitate further processing, we rearrange this sequence such that all class tokens are positioned at the beginming. This sequence é is then processed through several newly introduced transformer layers. These layers are designed to enhance interaction between the global-level and local-Ievel features contained in different images. This interaction is crucial as allows the model to gather the necessary information for ranking the alternatives. Upon completion of these layers, we collect the class tokens of alternatives kots = [1. These tokens are then processed through linear layers to generate the ranking prediction ù E R This prediction ý serves as an indicator of the ranking of the alternatives, providing quantitative measure of their relevance to the query sample.
Training objectives. Our ranking model is optimized with a composed objective as follow:
A max 1(y > 1,) I, + 6)) NouralNDDGG()(0..) MSE y). (1)
C 1J=1,i
Evere
Ereg
Cmargia
where Emargin denotes the pair-wise marginal ranking loss, L denotes list-wise Neural- NDCG [16). 5 and 7 represents the loss coefficients for two loss terms respectively. Lmargin and Creg act a similar role as the supervised contrastive loss in VPR, which encourages the model the predict higher score for better samples in each pair. Loort, the other hand, drive the model to learn inner relationship among more alternatives, thus leading to better ranking predictions.
3.2 Consistency-aware ranking aggregator
For this section, we will introduce how to obtain consistent global rank for the alternative set given the trained ranking model as introduced above.
Motivation. In the process of selecting in in-context examples, it is crucial to identify the most suitable candidates from the available pool. However, the ranking model we employ only provides partial ranking predictions on a subset instead of the full alternative set. A naive solution would be splitting the alternative set into non-overlapping subsets, which are then processed with the trained ranking model respectively. Then the top ranked samples from each subset are gathered together as new alternative set, which then repeatedly split and ranked again until there is only one sample.
While such method can provide reasonable results as we will show in the experiment, it can suffer from three main problems: (1) In total, (K) subsets with size & can be randomly sampled from an subsets, which is extremely alternative set with size K. However, the naive method only utilizes [
5

---

Algorithm Consistency-aware ranking aggregator
Input: Train set Agrain, query sample Zg, trained ranking models { alternative set size K.
1: Alternative set XR = top (sim( zq])
2: Initial preference matrix set ==
3: for rank-k model d% do
4: Build observation pool Nr. from RR
5: for randomly shuffled X2 from Xg do
6: Ri = UREx; 0x (x, xn))
7: Aggregate S from R2
8: S S=SUs
9: end for
10: end for
11: Aggregate global ranking as Eq.
12: return Top ranked sample.
limited and can result in selecting poor performing in-context examples. (2) Since this naive method iterative, the selection error would accumulate, leading to unbearably wrong prediction. (3) inefficient to fetch other top ranked samples such second third best ones with this method. Therefore a new method, which can directly provide global ranking prediction by fully utilizing information contained in the alternative set, can thus better serve the purpose of in-context example selection. To this end we propose the consistency-awar derive global ranking from the partial information, thereby making possible rank all candidates in relation to each other while guaranteeing global consistency.
ranking aggregator, which allows us to
Algorithm. Our method starts with gathering enough sufficient information from all alternatives. Given the query sample zg. alternative set X'n and a trained k-length ranking model ők. we can first build observation pool Xx ( where X2K denotes randomly shuffled variant of X'R- Then for each X1 we follow the naive ranking method to split it into non-overlapped k-length sequences and rank these sequences with @x. resulting in predicting further aggregated into preference matrix S R If the m-th alternative is favored the n-th alternative, we let St mn = 1 otherwise Stm = 1. If no predictions are related with m and n-th alternatives, then Stm O. S can then be transformed into an pair-wise indication set E = [(m, a)|S * 0}. In this way we can finally get preference matrix set S = {
Ri. The predictions are
Denoting the global ranking of Xn ranking. If we have the oracle ranking rk. then m-th candidate favored than n-th candidate we formulate the ranking problem as follows.
vector R in which higher score denotes higher
get r. r 0. As result,
min
Σ
(Tm- - In St St
(M.))EE
To get the solution reformulate Eq. aas Least Square problem by introducing transformation matrix D' IRI [ 1 each row of which is a sparse vector with only the two indices from the pairwise set E as and -1. Then we have the following formulation,
N
1
min
I 111
2N,
By solving Eq. we can directly get a proper global ranking r. Furthermore, multiple ranking models can be engaged in this process by extending S with preference matrices calculated from these different models, which can in turn enhance the ranking process with more comprehensive information. Compared with the naive ranking method, this method involves more partial observations from XR. while avoiding the accumulative error, thus being both effective and efficient.
Analysis. It is straightforward to the merits of the proposed method. First, compared with the naive strategy, solving Eq. 3 can directly lead to a complete ranking results, avoiding repetitive computation for more top ranked examples. Second, it is inevitable that the trained ranking model

---

wrongly predicts the ranking of some alternatives. Under such cireumstance, the naive strategy has now way fix such an error. On the contrary, since same alternative pair can result in different predictions by our list-wise ranker, the proposed algorithm as in Alg. | can take into account all contradicting predictions, thus leaving potential for rectifying the wrong predictions.
4 Experiments
4.1 Dataset and setting
Dataset. We follow MAE-VQGAN [1] and VPR 33 to adopt three tasks including foreground segmentation, single object detection and colorization. For foreground segmentation, Pascal-5' [19] is utilized which contains 4 data splits. We conduct experiments and report the mean intersection over union (mloU) on all splits together with the averaged mloU among these splits. For single object detection, Pascal VOC 2012 [5] is used. Images and predictions are processed the same as in MAE-VQGAN, in which mloU is adopted as metric. For colorization, we first sample 50000 training data from ILSVRC2012 2 18 training set. Then test set randomly sampled from the validation set of ILSVRC2012 is used to test the model with mean squared error (MSE) as metric.
Implementation Detail. Considering the training data size for each task, we adopt different sequence length for ranking. Specifically, we train rank-5 and rank-10 models for foreground segmentation and single object detection, while rank-3 and rank-5 models are trained for colorization. The training data is built by first selecting 50 most similar images from the whole training set for each image as its alternative set. Then we randomly sample 20 sequences with the required length from this alternative set for training. For all experiments we adopt DINO v2 [14] as feature extracter. AdamW W optimizer [12 is used with learning rate set as 5e-5 and batch size set as 64, We utilize 4 V100 gpus to cover all experiments.
Competitors. We choose three methods our competitors, including MAE VQQAN, VPR which contains UnsupPR and SupPR and prompt-Sell [2I]. which mainly facilitates an ensemble strategy. To fairly compare our methods with these previous ones, we report main results of our models both with and without the test-time ensemble.
4.2
Main results
The experiment results are shown in Tab. Note that we omit the result of our model with voting strategy for colorization since prompt-Self does not report this term and there is no fair competitors in this setting.
As in Tab. 1 for all three tasks, our model receives best performance on both variant. Specifically, when not using voting strategy, our model outperforms the strongest competitors SupPR by 2.84 in terms of average segmentation mIoU, while the superiority is consistent on other two tasks. When using voting strategy, our model receives about 4 higher mloU for segmentation and 2 higher mloU for detection, leading to better results than prompt-SelF, One would note that while SupPR was designed to learn a better metric than the naive visual similarity used for UnsupPR, its performance on colorization is exactly the same as UnsupPR. On contrary to that, our model makes a huge leap with 0.04 less MSE, which is not only much better than both UnsupPR and SupPR but also doubles the improvement these two methods have against the random strategy in MAE-VQGAN This can prove the effectiveness of our proposed ranking model for selecting in-context examples.
To further illustrate the advantage of Partial2Global,
visualize several samples for segmentation and detection in Fig. and Fig. 5) We find that in some cases the in-context examples selected by VPR can let MAE-VQGA VQQAN generate totally wrong results, especially when multiple objects are presented in the query. For example, in the first sample of Fig. 3 the query image contains two monitors, while the target label is only related to the left one. While both VPR and our method select images of monitors as in-context example, VPR results in 0 loU while prediction using our example much better. This may resulted from the spatial relation of the object of interest in the example images. In the example selected by VPR, the monitor is placed at the right, which is the same as the non-target monitor in the query image, thus leading to wrong guidance and prediction. Apart from that, VPR tends to select small objects for detection, as shown in Fig. 5 This can lead to failed detection even though the query objects are large enough for huamns to detect them. In comparison, method can generally select more proper in-context examples, thus enjoying better performance.
7

---

Table 1: Comparison of our method with previous in-context learning methods.
Seg, (mloU) f
Model
Det. (mloU)
Color. (MSE)
Fold-O
Fold-1
Fold-2
Fold-3
Avg.
MAF-VQGAN
28.66
30.21
27.81
23.55
27.56
25.45
0.67
UnsupPR
34.75
35.92
32.41
31.16
33.56
26.84
0.63
SupPR
37.08
38,43
34.40
32.32
35.56
28.22
0.63
Ours
38.81
41.54
37.25
36.01
38,40
30.66
0.58
prompt-SelF
42.48
43.34
39.76
38.50
41.02
29.83
Ours+voting
43.23
45.50
41.79
40.22
42.69
32.52
SupPR
31.86
2.54
1.59
2.48
16.17
29.29
30.55
Ours
Figure 3: Qualitative comparison between our method and VPR, specifically SupPR, in foreground segmentation. In each item we present the image grid in the same order as the input MAE- VQGAN, i.e. in-context example and its label in the first row, query image and its prediction in the second row. The loU is listed below each image grid.
4.3
Model analysis
To fully validate the effectiveness of each module in our method, we conduct series of ablation studies. For all ablation studies, we do not use voting strategy. The experiments contain both foreground segmentation and single object detection if specified.
Effectiveness of different modules. First of all we provide comparison among variants with different designs. Specifically, consider three models: (1) Rank-10 Naive: A rank-10 model is trained, of which the ranking predictions are directly used for example selection. (2) Rank-10 Aggr: After training the rank- 10 model and getting the ranking predictions, we use the proposed consisteney- aware ranking aggregator described in Sec. 3.2 to process these data, then we pick the top-ranked sample in the post-processed global ranking as in-context example. (3) (5,10( Aggr.: Our full model, in which the ranking prediction of both ranking models are mixed and processed with consistency-awee awrre ranking aggregator get the final global ranking. The results are presented in Tab. 2 We find that the naive ranking model enjoys 1.54 higher average mloU than SupPR, which supports our motivation of using ranking model instead of pair-wise contrastive learning model. Since ranking model can make better use of the training data, and comparing multiple samples enables model to discover the inner relationship between them, rather than simply judging if one sample is better than another one. Moreover, We find that using consistency-awart ranking aggregator to process the ranking predictions can lead to further improvement, which can enlarged by including results from rank-5 model. This is because the predictions of ranking models can be thought of as from different annotators. Even for single model, its predictions will not be consistent when given two sequences which share partial same samples. Therefore, simply utilizing the inconsistent predictions can lead to suboptimal results, and consistency-awant aware ranking aggregator helps regulate the predictions produce more globally consistent ranking results, thus having better results.
Is quality of in-context examples really aligned with visual similarity? basic claim in VPR is that the more similar an image is to the query image, the better it is as an in-context example. Nonetheless, VPR utilizes performance metrics instead of visual similarity for contrastive learning and provides better results, which makes one wonder if such claim is indeed solid. To check this

---

Table 2: Ablation study among different variants of our method.
Seg. (mloU) t
Model
Det. (mloU)
Fold-O
Fold-1
Fold-2
Fold-3
Avg.
SupPR
37.08
38.43
34.40
32.32
35.56
28.22
Rank-10 10 Naive
37.51
39.69
36.62
34.58
37.10
29.58
Rank-10 Aggr.
38.70
41.08
37.04
35.54
38.09
29.79
Rank- (5.10) Aggr.
38.81
41.54
37.25
36.01
38.40
30.66
out, we take look at the best performing in-context examples selected by our method. The results are shown in Fig. a) and (b). We first visualize the correlation between visual similarity, which computed by extracting CLIP features for each image and then calculating and averaging the cosine similarity between the alternatives and query, and mloU on segmentation for both VPR and our method. is clear that visual similarity can be a basic heuristic for choosing visual in-context examples. Generally well performing examples appear to share high similarity with query samples. On the other hand, there are also large amount of failure cases with high similarity, which indicates ground principle should be much more complex than visual similarity. Based on this conclusion, further visualize several cases in Fig. 4c) selected by our method, which better shows that one should consider more than visual similarity and stress other factors such object size, spatial position. On the opposite, we find that the background similarity, which can contribute significantly to the visual similarity, is hardly related to the quality of in-context examples.
45 22,
61.78, 0:84
$1.01.0.66
62.25, 0.87
$1.20 0.70
48 0.71
stada anilaty
1,64, 0.08
1,233
0.30
1407,
Figure 4: (a) Scatter plot of visual similarity against IoU for VPR on segmentation. (b) Scatter plot of visual similarity against IoU for our method on segmentation. (c) Visualization of several cases with uncorrelated visual similarity and IoU. The first row presents samples with low similarity but proper in-context performance. The second row presents samples with high similarity but poor in-context performance. Captions below each image grid denote IoU and visual similarity sequentially.
Robustness among different backbones. We compare our model with two variants with other pretrained feature extractor: CLIP ViT-L/14 and DINO v1. The results are shown in Tab. 3 First, directly using ranking prediction from our ranking models through the naive ranking strategy performs better than VPR no matter which backbone is utilized. VPR also conducted such an experiment to try their method with different backbones such as CLIP, EVA and supervised ViT, but none of those results can outperform ours, which illustrates the efficacy of our design of ranking model against the contrastive learning based model. Second, adopting consistency-aware ranking aggregator consistently improves the naive ranking method by nearly 1-2 mloU for both segmentation and detection, which compatible with the results in Tab. - showing the robustness of proposed method among different backbone choices. Third, the in-context performance is not totally correlated with the capacity of backbones. is commonly known that DINO v2, as an improved version of DINO vI with more comprehensive objective functions and more data. should be equipped with higher capacity. However, we find that DINO v2 performs worse than DINO vl on segmentation whether naive ranking or consistency-awe aware ranking aggregator is adopted. We think such results can be attributed to the different learning difficulty when using these backbones, and would be interesting to have further research the impact of backbones for visual in-context learning.
Consistent optimality ranking results. Apart from the strong performance of our method, we further find that the proposed consistency-awarr ranking aggregator can interestingly introduce consistent optimality to the ranking results. That is say, apart from the top ranked sample, the second best and third best samples should also be generally better than other samples in the alternative

---

Table 3: Ablation study among different backbones of our method.
Seg. (mloU) 1
Backhone
Strategy
Det. (mIoU)
Fold-O
Fold-1
Fold-2
Fold-3
Avg.
Naive
37.37
40.11
36.84
33.88
37.05
29.69
CLIP
Aggr.
38.58
41.34
37.66
35.91
38.37
30.79
Naive
38.78
40.02
36,92
35.12
37.71
28.03
DINOv1
Aggr.
39.25
42.27
38.45
36.77
39.19
29.19
Naive Aggr.
37.51
39.69
36.62
34.58
37.10
29.58
DINOv2
38.81
41.54
37.25
36.01
38.40
30.66
Table 4: Ablation study among different example selection strategies of our method.
Seg. (mloU) f
Strategy
Det. (mIoU)
Fold-O
Fold-1
Fold-2
Fold-3
Avg.
#1 rank
38.81
41.54
37.25
36.01
38.40
30.66
#2 rank
38.13
41.66
37.62
35.35
38.19
30.76
#3 rank
38.66
41.08
37.36
35.91
38,25
30.61
top2 fusion
39.08
42.61
38.17
36.67
39.13
30.16
top3 fusion
40.07
42.48
38.77
37.61
39.73
31.85
top5 fusion
40.12
42.59
39.09
37.28
39.77
32.08
set as in-context example. To show that we test two strategies: (1) directly using samples ranked second and third as in-context examples and (2) adopting a simple late fusion method to average predictions generated with 2, 3, 5 top ranked samples. The results are shown in Tab. 4 One can find that using top-3 best ranked sample have comparable results. For segmentation, samples only have 0.21 lower mIoU than #1 rank samples, and #3 rank samples are even slightly better than #2 rank samples, which evidences the consistent optimality. The results are consistent for detection. Moreover, fusing 2 or 3 samples together leads to 0.73 and 1.43 higher average mloU, which also supports the consistent optimality brought by the consistency-awart ranking aggregator. Such improvement saturates when using 5 samples, indicating that samples ranked 5-th or later are significantly worse and cannot provide more useful information for the in-context inference.
#2 rank
5 Conclusion and discussion
This paper proposes novel pipeline for in-context example selection in Visual In-Context Learning (VICL). Specifically, we design a transformer- based ranking model, which can provide desirable ranking predictions by well utilizing the inner relationship among alternatives. The ranking results are further aggregated by the proposed consistent-awee aware ranking aggregator, which achieves desirable global ranking prediction by transforming the problem into least square problem. Our method receives state-of-the-art performance on three different tasks including foreground segmentation, single object detection and image colorization, showing great potential of improving VICL researches.
Limitations. While our proposed method can provide good in-context examples, the basic perfor- mance still highly relies the quality of the in-context learning model. The MAE-VQGAT used in this paper is rather an early stage trial for visual in in-context learning, with many limitations in terms of its structures and applications. We believe that our model can further help other stronger visual in-context learning methods, leading to better in-context performance.
Broader impacts. Our work will not lead to significant negative social impacts. One main problem is that if the data used for in-context learning contains biases, the models trained on this data may also exhibit these biases, leading to unfair or discriminatory outcomes. This is particularly concerning in areas like facial recognition or predictive policing, where biased models could disproportionately affect certain groups. Solving such problems would be large future rescarch topic.
10

---

Acknowledgments and Disclosure of Funding
Y. Yao was in part supported by the HKRGC GRF-1630832 and the NSFC/RGC Joint Research Scheme Grant N HKUST635/20. The computations in this research were performed using the CFFF platform of Fudan University. This project is supported by Al for Science Foundation of Fudan University.
References
[1] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005-25017, 2022.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1 177--191,, 2020.
[3] Christopher Burges, Robert Ragno, and Quoc Le. Learning rank with nonsmooth cost functions. Advancey neural information processing systems, 2006.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf-supe vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9600, 2021.
[5] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98- 136, 2015.
[6] Zhonghin Fang, Xiangtai Li, Xia Li, Joachim M Buhmann, Chen Change I Loy, and Mengyuan Liu, Explore in-context learning for 3d point cloud understanding. Advances in Newral Information Processing Systems, 36, 2024.
[7] Roce Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv preprint arXiv:2310. 15916, 2023.
[8] Xiaoye Jiang, Lek-Heng Lim, Yuan Yao, and Yinyu Ye. Statistical ranking and combinatorial hodge theory. Mathematical Programming, 127(1):0032244, 2011.
[9] Pawan Kumar, Dhanajit Brahma, Harish Karnick, and Piyush Rai. Deep attentive ranking networks for learning to order sentences. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 81 15-8122, 2020.
[10] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[11] Jiduan Liu, Jiahao Liu, Qifan Wang, Jingang Wang, Wei Wu, Yunsen Xian, Dongyan Zhao, Kai Chen, and Rui Yan. Rankese: Unsupervised sentence representations learning via learning rank. arXiv preprint arXiv:2305. 16726, 2023.
[12] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint ax1iv17711.01101.2011 2017.
[13] Nagarajan Natarajan, Inderjit Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with noisy lahels. Advances in neural information processing systems, 26, 2013.
[14] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719. 2023.
[15] Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-Im: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint 10X2:22305.12295 2023.
[16] Przemyslaw Pobrotyn and Radoslaw Bialobrzeski. Neuralndog: Direct optimisation of a ranking metric via diffenentiable relaxation of sorting, arXiv preprint arXiv:2102.07831, 2021.
11

---

[17] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763, PMLR. 2021.
[18] Olga Russakovsky. Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Audrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:21 252, 2015.
[19] Amirreza Shaban. Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint (tt:::7709034//0 2017.
[20] Yiyuan She and Art Owen. Outlier detection using nonconvex penalized regression. Journal of the American Statistical Association, 106(494):226666 639, 2011.
[21] Yanpeng Sun, Qiang Chen, Jian Wang. Jingdong Wang, and Zechao Li. Exploring effective factors for improving visual in-context learning. arXiv preprint 2.Xiv:330.0044:: 2023.
[22] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[23] Maksims N Volkovs and Richard S Zemel. Boltzrank: Proceedings of the 26th Annual International Conference on Machine Learning, pages 1089-1096, 2009.
learning to maximize expected ranking gain. In
[24] Xinlong Wang. Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 6830 2023.
[25] Yikai Wang, Yanwei Fu, and Xiawei Sun. Knockofls-spr: Clean sample selection in learning with noisy labels. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[26] Yikai Wang, Xinwei Sun. and Yanwei Fu. Scalable penalized regression for noise detection in learning with noisy labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 346-355, 2022.
[27] Yikai Wang. Chengming Xu, Chen Liu, Li Zhang, and Yanwei Fu. Instance credibility inference for few-shot learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12836 12845, 2020.
[28] Yikai Wang, Li Zhang, Yuan Yao, and Yanwei Fu. How to trust unlabeled data? instance credibil- ity inference for few-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[29] Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang. Mingyuan Zhou, et al. In-context learning unlocked for diffusion models. Advances in Neural Information Processing Systems, 36:8542-8566 2023.
[30] Qianqian Xu, Qingming Huang, Tingting Jiang, Bowei Yan, Weisi Lin, and Yuan Yao. Hodgerank on random graphs for subjective video quality assessment. IEEE Transactions Multimedio, 14(3):844 857, 2012.
[31] Qianqian Xu, Jicchao Xiong, Xi Chen, Qingming Huang, and Yuan Yao. Hodgerank with information maximization for crowdsoureed pairwise ranking aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[32] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36, 2024.
[33] Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305. /2740, 2023.
[34] Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. Visual in-context learnigg for large vision-language models. arXiv preprint arXiv:2402. 15574, 2024.
12

---

A More details
NeuraINDCG. The list-wise ranking loss NeuraINDCG adopted in this paper is differentiable approximation of Normalised Discounted Cumulative Gain (NDCG). We simply quote the definition here from []6 for better understanding. Specifically, for æ denoting a sample and y denoting its score, NDCG can be calculated as
2
DCG(E, loga (1 + (1))
(4)
NDCG(TT, DCG(T".))
where f denotes the predicted ranking, is the ground truth ranking regarding score y. Neural- NDCG works by substituting the discontinuous sorting operator with NeuralSort, which results in an approximated permutation matrix
(n + 1 21}s A,1
Psort (s) (T) softmas
(6)
T
where Ag = |s, syl. denotes vector filled with value 1, t is a temperature parameter. Then the NeuraINDCG can be calculated as
n scale P22 1) log2(1 + x x1))
DCG(F)(T,V)
DCG(T)(T,W) DCG
D
(8)
where scale(.) is Sinkhorn scaling.
SupPR
Z
0
10.88
0
0
0
17.99
31.27
28.32
Oura
79.48
89.90
75.27
69.81
66.32
75.92
87.26
83.44
SupPR
21.42
26.41
0
29.86
Ours
52.95
73.11
49.75
49,00
71.55
45.03
74.79
44,57
SupPR
Ours
44.43
67.63
46.96
44.00
50.00
46.82
37.17
43.72
Figure 5: Qualitative comparison between our method and VPR, specifically SupPR, in single object detection. For simplicity we present the bounding boxes on images instead of showing the image grids. In each item the left image denotes the in-context example and the right one denotes the query.
B Additional ablation study
Transferability of Partial2Glohal. To test the transferability the prompt selection method and further reveal the potential of our method, we can add the demonstration suggested by conducting
13

---

the following experiment for both SupPR and our method: we use models trained on each fold of segmentation and upply it to other folds. The results are shown in T We find that while both two methods degrade in the transfer learning setting, our method still outperforms SupPR in general. This interesting results indicate that, the training data size of each fold is insuifficient for training a robust and generalizable ranking model. On the other hand, this again indicates that prompt selection for VICL cannot be simply based on visual similarity, as claimed our paper.
Table 5: Cross-fold performance of our
Table 6: Cross-fold performance of SupPR on segmentation task.
method on segmentation task.
Sturce/Target Fold
0
2
3
Source/Target Fold
2
0
36,38
32.63
30.90
0
35.46
32,44
30.95
35.74
32,94
31.32
1
34.92
32.96
31.03
34.16
36.16
30.44
2
34.71
36.48
30.08
34.28
35,93
32.98
3
34.01
35.83
32.15
Efficiency of our method. One would ask if the proposed method would suffer from poor efficiency, compared with SupPR. In general, the usage of list ranker and the ranking aggregation will inevitably introduce additional computational cost, while the increased complexity during inference affordable under common circumstances. Specifically, we provide the training and inference time cost as follows. (1) The training of list-wise ranker the colorization task, which contains about 500000 ranking sequences, takes about 10 hours on 8 V100s. Once the model is trained it can be directly used for any other queries with the same ranking criteria the training task without any further finetuning.
(2) During inference on one V100 gpu, our proposed pipeline requires about 1.17s to rank 50 alternatives for each query sample in the complete process, including feature extracting (0.3s), sub-sequence ranking with list-wise ranker (0.8s), and ranking aggregation (0.07s). Note that some techniques could be utilized accelerate this process. For example, when we prepare the extracted features in advance (which is reasonable given the candidate set can be prepared in advance), we can skip the feature extraction stage and reduce the time cost by 0.3s. With engincering works, the inference time cost can be further reduced. The detailed inference speed (feature extraction included) given different alternative set size presented as Tab.
Table 7: Inference speed with different alternative set size.
alternative set size
inference time for each query (s)
25
1.03
50
1.17
100
1.40
Upper bound of selecting different in-context prompts. Another interesting problem is to examine the 'upper bound' of performance by adopting different in-context prompts. To further provide insight to this task, we try to examine an upper bound: directly testing all alternatives for each query in the segmentation task and presenting the best IoU in Tab. [ While our proposed method is much better than SupPR, it still leaves great potential for better performance, which we will take as future works.
Table 8: Oracle in-context learning performance segmentation.
fold0
fold1
fold2
fold3
SupPR
37.08
38.43
34.40
32.32
Ours
38.81
41.54
37.25
36.01
best iou among 50 alternative (oracle)
48.75
52.62
49.75
49.03
Sensitivity test for hyper-parameters. In general, our method is robust against changes of hyper- parameters. To show this, we try the suggestions to conduct ablation studies on these two hyper- parameters, whose results are presented in the following table. For delta which denotes the margin, using 0 margin leads to worse results while larger margins would be better for learning ranking models. For tau which is the temperature coefficient in NeuraINDCG, we simply use the best setting tau=1 from the original paper. As can be seen in Tab. 9 using smaller temperature would not lead to better results. Nonetheless, all hyperparameter settings enjoy better performance than SupPR, validating the effectiveness of our method.
14

---

Table 9: Ablation study for
2
T= 0.01
т 0.1
MSE 0.594
0.588
0.608
0.592
Impact of alternative set size on performance. Our choice of alternative set is the same VPR, Le. selecting 50 most visually similar samples hased on CLIP features. Typically, this choice is good enough shown in previous works like VPR. Here, to further investigate the impact of data quality, we thus tried the suggestion to test our method on all folds of segmentation task with 25 or 100 alternatives for each query. The results are shown in Tab. 10 One can find that our method is also very robust against different sizes of alternative set.
Table 10: Ablation study for different alternative sizes.
set size
fold0
fold1
fold2
fold3
25
38.48
41.82
37.14
35.60
50 (main)
38.81
41.54
37.25
36.01
38.81
41.80
37.90
36.00
Effectiveness of different terms in our proposed loss. We have conducted an additional ablation study compare models trained for colorization without each loss term. The results are shown in Tab. [1] In general all three loss terms contribute the final performance, with Lortt plays the most important role.
Table 11: Ablation study for loss terms.
w/o Exort
w/o Emargin
w/o Linn
full loss
MSE
0.601
0.595
0.585
0.583
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract claims that we propose novel pipeline for visual in-context learning example selection, which is consistent with our main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have analyzed the limitation of our main in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
Our proposed method does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided experiment details in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We have provided the data source Sec. 4.1 The corresponding code also available now.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We cannot provide error bar for some competitors. Besides, the current results are sufficient show the efficacy of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pp....aaoo....//op//////a
Answer: [Yes]
Justification:
Our research focuses general visual tasks, which totally conforms the Ethics
Code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to the Sec. 5 in the main paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have not released our model weight yet.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all used public datasets and pre-trained models in Sec. 4.1
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We only use objective experiments in our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We only use objective experiments in our paper.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "ZIArzhbOXi", "reviewer": "Reviewer_crFX", "summary": "This paper studies the demonstration retrieval mechanism for visual in-context learning. The authors propose Partial2Global which uses a transformer-based ranker and a consistency-aware aggregator to find the optimal demonstration. Experiments show that Partial2Global outperforms existing methods in tasks like segmentation and object detection, setting new state-of-the-art results.", "questions": "See the Weaknesses.", "limitations": "N/A.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The motivation is clear and reasonable.\n\n3. The proposed method is interesting and novel. Exploring efficient and effective global ranking in place of the previous local information and using least-squares fusion to mitigate iterative loss.\n\n4. The ablation studies are adequate, revealing the effectiveness of global information, the efficacy of the fusion method, and that similar images do not necessarily yield better contextual effects. It also identifies defects in samples retrieved by methods like VPR (e.g., frequently retrieving smaller objects in detection tasks, which results in poorer VICL performance).", "weaknesses": "1. Since training exclusively on a single dataset is limiting, I would appreciate a demonstration of transfer learning performance to prove its adaptability and universality across a broad range of data.\n\n2. It would be beneficial to analyze the efficiency of your model to the size of the retrieval set, as complexity is a crucial aspect.\n\n3. It would be insightful to examine the \"upper bound\" of performance, such as by concatenating each demonstration with a query to see what the maximum Intersection over Union (IoU) is, and to assess how much room there is for improvement from this upper bound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Since training exclusively on a single dataset is limiting, I would appreciate a demonstration of transfer learning performance to prove its adaptability and universality across a broad range of data.**\n\nA1: Thank you for the suggestion. We would like to highlight that the current experiment results can sufficiently validate the efficacy of our method. Nevertheless, to test the transferability of the prompt selection method and further reveal the potential of our method, we can add the demonstration suggested by conducting the following experiment for both SupPR and our method: we use models trained on each fold of segmentation and apply it to other folds. The results are shown in the following two tables. We find that while both two methods degrade in the transfer learning setting, our method still outperforms SupPR in general.   \nThis interesting results indicate that, the training data size of each fold is insuifficient for training a robust and generalizable ranking model. On the other hand, this again indicates that prompt selection for VICL cannot be simply based on visual similarity, as claimed in our paper.  \n\nOurs\n\n| source/target fold | 0 | 1 | 2 | 3 |\n| :---- | :---- | :---- | :---- | :---- |\n| 0 | — | 36.38 | 32.63 | 30.90 |\n| 1 | 35.74 | — | 32.94 | 31.32 |\n| 2 | 34.16 | 36.16 | — | 30.44 |\n| 3 | 34.28 | 35.93 | 32.98 | — |\n\nSupPR\n\n| source/target fold | 0 | 1 | 2 | 3 |\n| :---- | :---- | :---- | :---- | :---- |\n| 0 | — | 35.46 | 32.44 | 30.95 |\n| 1 | 34.92 | — | 32.69 | 31.03 |\n| 2 | 34.71 | 36.48 | — | 30.08 |\n| 3 | 34.01 | 35.83 | 32.15 | — |\n\n**Q2: It would be beneficial to analyze the efficiency of your model to the size of the retrieval set, as complexity is a crucial aspect.**\n\nA2: Thank you. Please refer to the general response. Generally, we can summarize that the usage of list-wise ranker and the ranking aggregation will inevitably introduce additional computational cost, while the increased complexity during inference is affordable in our experiments.\n\n**Q3: It would be insightful to examine the \"upper bound\" of performance, such as by concatenating each demonstration with a query to see what the maximum Intersection over Union (IoU) is, and to assess how much room there is for improvement from this upper bound.**\n\nA3: Thank you for this suggestion. We want to first highlight that our experiment results can already validate the efficacy of our method against the competitor as a powerful in-context prompt selection method. To further provide insight to this task, we try to examine an upper bound: directly testing all alternatives for each query in the segmentation task and presenting the best IoU in the following table. While our proposed method is much better than SupPR, it still leaves great potential for better performance, which we will take as future works.\n\n|  | fold0 | fold1 | fold2 | fold3 |\n| :---- | :---- | :---- | :---- | :---- |\n| SupPR | 37.08 | 38.43 | 34.40 | 32.32 |\n| Ours | 38.81 | 41.54 | 37.25 | 36.01 |\n| best iou among 50 alternative (oracle) | 48.75 | 52.62 | 49.75 | 49.03 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u9hEiX4yaG", "reviewer": "Reviewer_dHhp", "summary": "This paper addresses Visual In-Context Learning (VICL), which uses examples to help models learn new tasks. The main challenge is choosing the best prompt to improve learning and prediction. The authors introduce Partial2Global, a new method to find the best examples for each query. This method uses a transformer-based ranker to compare alternatives and a ranking aggregator for consistency. Experiments show that Partial2Global outperforms other methods in tasks like segmentation, object detection, and colorization, setting new performance standards.", "questions": "Please see Weaknesses.", "limitations": "The authors address the limitations and potential negative societal impacts of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The proposed method, Partial2Global, introduces a novel framework for in-context example selection that addresses the fundamental challenge in Visual In-Context Learning (VICL) with a unique transformer-based list-wise ranker and a consistency-aware ranking aggregator.\n\n2.The authors have conducted extensive experiments across multiple tasks, including foreground segmentation, single object detection, and image colorization. This robust evaluation provides strong evidence of the method's effectiveness and generalizability.\n\n3.  The paper is well-written and easy to follow, with complex ideas clearly explained. In particular, Figure 2 offers a clear and detailed comparison of existing methods, helping to contextualize the contributions of the proposed approach.", "weaknesses": "1.The paper does not include ablation studies on the critical parameters, such as δ and τ. These studies are essential for understanding how variations in these parameters affect the model's performance. Without this information, it is difficult to assess the robustness and sensitivity of the model to changes in these parameters.\n\n2.The paper introduces heuristic methods for ranking in-context examples using a list-wise ranker and a consistency-aware ranking aggregator. However, there is no analysis provided on the computational complexity and time cost associated with these methods. Given the potential for high computational demands, especially with large datasets, it is important to understand the scalability and practicality of these approaches.\n\n3.Figure 4 of the paper presents results related to visual similarity, but it lacks a clear explanation of how this similarity is computed. The method or formula used to calculate visual similarity is not provided, which is crucial for the reproducibility of the results and for understanding the underlying methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The paper does not include ablation studies on the critical parameters, such as δ and τ (NDCG). These studies are essential for understanding how variations in these parameters affect the model's performance. Without this information, it is difficult to assess the robustness and sensitivity of the model to changes in these parameters.**\n\nA1: In general, our method is robust against changes of hyper-parameters. To show this, we  try the suggestions to conduct ablation studies on these two hyper-parameters, whose results are presented in the following table. For delta which denotes the margin, using 0 margin leads to worse results while larger margins would be better for learning ranking models. For tau which is the temperature coefficient in NeuralNDCG, we simply use the best setting tau=1 from the original paper. As can be seen in the following table, using smaller temperature would not lead to better results. Nonetheless, all hyperparameter settings enjoy better performance than SupPR, validating the effectiveness of our method.\n\n|  | MSE |\n| :---- | :---- |\n| $\\delta$=0 | 0.594 |\n| $\\delta$=2 | 0.588 |\n| $\\tau$=0.01 | 0.608 |\n| $\\tau$=0.1 | 0.592 |\n\n**Q2: The paper introduces heuristic methods for ranking in-context examples using a list-wise ranker and a consistency-aware ranking aggregator. However, there is no analysis provided on the computational complexity and time cost associated with these methods. Given the potential for high computational demands, especially with large datasets, it is important to understand the scalability and practicality of these approaches.**\n\nA2: Thank you. Please refer to the general response. We give a thorough analysis. Generally, we can summarize that the usage of list-wise ranker and the ranking aggregation will inevitably introduce additional computational cost, while the increased complexity during inference is affordable in our experiments.\n\n**Q3: Figure 4 of the paper presents results related to visual similarity, but it lacks a clear explanation of how this similarity is computed. The method or formula used to calculate visual similarity is not provided, which is crucial for the reproducibility of the results and for understanding the underlying methodology.**\n\nA3: Thank you. As illustrated in L310, we use commonly used CLIP to extract features for each image, and then we calculate the cosine similarity between the alternatives and query, which are then averaged. We will add the details to our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ttS0IDVHUk", "reviewer": "Reviewer_PwDC", "summary": "This paper addresses the fundamental problem in Visual In-Context Learning (VICL), which is how to select the best prompt. Specifically, it focuses on the ranking problem. The paper proposes an algorithm called Partial2Global, which conducts an in-context example selection framework to find the global optimal prompt. The effectiveness of the proposed algorithm is demonstrated through several tasks, such as segmentation, object detection, and colorization.", "questions": "Please see the weakness part.", "limitations": "Please see the weakness part.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The algorithm is simple yet intuitive for finding the optimal global ranking using a list-wise ranker. It adopts a consistency-aware ranking aggregator.\n\n(2) The performance is evaluated across various tasks, showing improved results in those tasks.", "weaknesses": "The method seems to incur additional costs due to the ranking process. Is there any further analysis on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The method seems to incur additional costs due to the ranking process. Is there any further analysis on this?**\n\nA1: Thank you. Please refer to the general response. Generally, we can summarize that the usage of list-wise ranker and the ranking aggregation will inevitably introduce additional computational cost, while the increased complexity during inference is affordable in our experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dsYfBHgBVT", "reviewer": "Reviewer_dbja", "summary": "The paper proposes a framework called Partial2Global for ranking in-context examples in visual language models. This method uses a transformer-based list-wise ranker and a consistency aware ranking aggregator to approximate the global optimal prompt selection. The framework is validated through experiments on tasks such as foreground segmentation, single object detection, and image colorization, demonstrating its effectiveness over existing methods.", "questions": "See weakness part", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1：In the analysis, a relatively comprehensive examination was conducted on the sensitivity of different components and various backbones to the order of ICL selection, demonstrating the importance of reasonable prompt ranking for ICL in segmentation tasks.\n2：The writing and expression are relatively good.\n3：The reasons why sample ranking in ICL may affect the final outcome are analyzed in this paper.", "weaknesses": "1.The workload is relatively low. I think this work should also explore the impact of models with a larger number of shots on ICL. Additionally, it should investigate the impact of data quality in the alternative dataset, and include the importance of different parts of the loss in the ranking model during ablation experiments in formula (1).\n2.The conclusions presented in some charts are not intuitive. For example, in Figure 4, can the similarity of our method and the visual similarity of VPR be displayed in the same chart?\n3.Compared to previous methods, although it achieves a much better level, the improvement is relatively small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The workload is relatively low. I think this work should also explore the impact of models with a larger number of shots on ICL. Additionally, it should investigate the impact of data quality in the alternative dataset, and include the importance of different parts of the loss in the ranking model during ablation experiments in formula (1).**\n\nA1: (1) Larger number shots:Thanks, yes, we had tried many number of shots in our experiments, which are solid enough to support our contributions in generally. And we further add more number of shots in rebuttal. Particularly, our choice of number of shot can be referred to previous methods such as MAE-VQGAN and VPR. As for multi-shot experiment, we in this paper present two trials of multi-shot experiments. First, in Tab.1, we adopt a voting strategy to merge predictions of different permutations of each alternative-query pair. Second, in Tab.4, we present multi-alternative fusion. Both results indicate the efficacy of our method. To further verify the role of multi-alternative fusion, we extend experiments in Tab.4 to top-7 and top-10 fusion. The results are shown in the following table. We can find that the results are consistent with the original conclusion, where the performance is nearly saturated with 5 alternatives. This again supports that the experiments in our paper well-supports our claim. \n\n| \\#shot | fold0 | fold1 | fold2 | fold3 | Avg |\n| :---- | :---- | :---- | :---- | :---- | :---- |\n| top2 fusion | 39.08 | 42.61 | 38.17 | 36.67 | 39.13 |\n| top3 fusion | 40.07 | 42.48 | 38.77 | 37.61 | 39.73 |\n| top5 fusion | 40.12 | 42.59 | 39.09 | 37.28 | 39.77 |\n| top7 fusion | 40.14 | 42.36 | 39.01 | 37.32 | 39.71 |\n| top10 fusion | 40.14 | 42.43 | 39.09 | 36.66 | 39.58 |\n\n(2) Impact of data quality. Our choice of alternative set is the same as VPR, i.e. selecting 50 most visually similar samples based on CLIP features. Typically, this choice is good enough as shown in previous works like VPR. Here, to further investigate the impact of data quality, we thus tried the suggestion to test our method on all folds of segmentation task with 25 or 100 alternatives for each query. The results are shown in the following table. One can find that our method is also very robust against different sizes of alternative set.\n\nalternative set size\n\n| set size | fold0 | fold1 | fold2 | fold3 |\n| :---- | :---- | :---- | :---- | :---- |\n| 25 | 38.48 | 41.82 | 37.14 | 35.60 |\n| 50(main) | 38.81 | 41.54 | 37.25 | 36.01 |\n| 100 | 38.81 | 41.80 | 37.90 | 36.00 |\n\n(3) Different parts of the loss. Thanks. We have conducted an additional ablation study to compare models trained for colorization without each loss term. The results are shown in the following table. In general all three loss terms contribute the final performance, with L\\_sort plays the most important role.\n\nLoss terms\n\n| colorization | MSE |\n| :---- | :---- |\n| w/o L\\_sort | 0.601 |\n| w/o L\\_margin | 0.595 |\n| w/o L\\_mse | 0.585 |\n| full loss | 0.583 |\n\n**Q2: The conclusions presented in some charts are not intuitive. For example, in Figure 4, can the similarity of our method and the visual similarity of VPR be displayed in the same chart?**\n\nA2: Thank you, but as analyzed in L310-L314, visualization in Fig.4(a,b) is used to show that visual similarity can only act as basic heuristic for in-context example selection. Since we do not intend to compare results in these two figures, presenting results for two methods respectively is enough to support our claim. We will clarify this in our paper.\n\n**Q3: Compared to previous methods, although it achieves a much better level, the improvement is relatively small.**\n\nA3: For better understanding we summarize the improvement of both our method and SupPR against UnsupPR in the following table. Concretely, for segmentation and detection improvement of our method against SupPR is comparable to that of SupPR against UnsupPR. For colorization SupPR cannot improve based on UnsupPR while our method improves it by 0.05 MSE. These results are significant enough to validate the effectiveness of our proposed method.\n\n| Model | Seg-fold0 | Seg-fold1 | Seg-fold2 | Seg-fold3 | Det. | Color. |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| SupPR | \\+2.33 | \\+2.51 | \\+1.99 | \\+1.16 | \\+1.38 | \\-0.00 |\n| Ours | \\+4.06 | \\+5.62 | \\+4.84 | \\+4.85 | \\+3.82 | \\-0.05 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for all of your time to write valuable and constructive comments. Your feedback will definitely assist us in enhancing the quality of our paper, and thus we are committed to incorporating these suggestions in our revision process. Meanwhile, we feel encouraged that the reviewers find our paper well-written (Reviewer crFX, dHhp and dbja), our method novel and effective (Reviewer crFX, dHhp, PwDC), and our experiments solid and comprehensive (Reviewer crFX, dHhp and PwDC). Your support means a lot to us\! At this juncture, we would like to illustrate the general concern of training and testing efficiency of our method.

In general, the usage of list-wise ranker and the ranking aggregation will inevitably introduce additional computational cost, while the increased complexity during inference is affordable under common circumstances. Specifically, we provide the training and inference time cost as follows.

1) The training of list-wise ranker on the colorization task, which contains about 500000 ranking sequences, takes about 10 hours on 8 V100s. Once the model is trained it can be directly used for any other queries with the same ranking criteria as the training task without any further finetuning.   
2) During inference on one V100 gpu, our proposed pipeline requires about 1.17s to rank 50 alternatives for each query sample in the complete process, including feature extracting (0.3s), sub-sequence ranking with list-wise ranker (0.8s), and ranking aggregation (0.07s). Note that some techniques could be utilized to accelerate this process. For example, when we prepare the extracted features in advance (which is reasonable given the candidate set can be prepared in advance), we can skip the feature extraction stage and reduce the time cost by 0.3s. With engineering works, the inference time cost can be further reduced. The detailed inference speed (feature extraction included) given different alternative set size is presented as follows

| alternative set size | inference time for each query (s) |
| :---- | :---- |
| 25 | 1.03 |
| 50 | 1.17 |
| 100 | 1.40 |

In response to the reviewers' comments, we have thoroughly reviewed our paper, performed additional experiments, and prepared a comprehensive response. We will improve the manuscript according to your comments. We hope that our paper adequately addresses your concerns. We kindly look forward to your recognition.

Best regards.

Meta-review: The paper received two Weak Accept and two Borderline Accept responses. The AC read the paper, rebuttal, and discussions, and agree with the reviewers' recommendations. It is also encouraged that the authors incorporate reviewers' comments in their final version. Congratulations to the acceptance!
Decision: Accept (poster)