Towards Global Optimal Visual In-Context Learning Prompt Selection

Authors: chengming  xu
chen  liu
yikai  wang
yuan  yao
yanwei  fu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 08:09:49

Abstract: Visual In-Context Learning (VICL) is a prevailing way to transfer visual foundation models to new tasks by leveraging contextual information contained in in-context examples to enhance learning and prediction of query sample. The fundamental problem in VICL is how to select the best prompt to activate its power as much as possible, which is equivalent to the ranking problem to test the in-context behavior of each candidate in the alternative set and select the best one. To utilize more appropriate ranking metric and leverage more comprehensive information among the alternative set, we propose a novel in-context example selection framework to approximately identify the global optimal prompt, i.e. choosing the best performing in-context examples from all alternatives for each query sample. Our method, dubbed Partial2Global, adopts a transformer-based list-wise ranker to provide a more comprehensive comparison within several alternatives, and a consistency-aware ranking aggregator to generate globally consistent ranking. The effectiveness of Partial2Global is validated through experiments on foreground segmentation, single object detection and image colorization, demonstrating that Partial2Global selects consistently better in-context examples compared with other methods, and thus establish the new state-of-the-arts.

Main paper: 
 introduction: 
 Foundation models like those used in AIGC, such as GPT and Gemini, play a big role in its success. Different ways to fine-tune these models are being suggested to use them more effectively. One popular idea is called Visual In-Context Learning (VICL), which is a way to teach Visual Foundation Models (VFMs) using context from images. This helps the models learn better and make more accurate 'guesses' based on what they see. Basically, VICL gives the models prompt examples to learn from, like pairs of images and labels, termed in-context examples, based on which query samples are inferred. This mimics how humans learn with the guidance. Using this method, VFMs can take lots of different tasks, like understanding point clouds [6] , editing images [29] , as well as multi-modal inference [34] , even if they were not trained specifically for those tasks.The main challenge of VICL is picking the best in-context example that matches the query images, so the VFM can perform well. Empirically, we have found that randomly picking an example does not always work well, while a carefully selected example can really boost performance, as observed in Figure 1 : Qualitative comparison between our method and VPR, specifically SupPR, in foreground segmentation. In each item we present the image grid in the same order as the input of MAE-VQGAN, i.e. in-context example and its label in the first row, query image and its prediction in the second row. The IoU is listed below each image grid.previous works [32, 21] , and visualized comparison in Fig. 1 . Typically, VICL has a set of options to choose from (like the training data). We can think of picking the best in-context example as a ranking problem -we want to choose the one that helps the most. Ideally, we would like to test each option with a new image, rank how well they work, and then pick the best one. However, doing that for every image is not practical. So, one of the biggest challenges in VICL is figuring out how to do this ranking without actually testing each option.There are a few key challenges in choosing the best prompt for VICL. (1) Choosing the Right Metric: Since we cannot directly test how well a prompt works on a specific image, we need to find a different way to measure it. Previous methods like VPR [32] mainly use two metrics from ranked observations in the training set: visual similarity and pair-wise relevance instantiated through contrastive learning. However, visual similarity does not always work well as in our experiments. The contrastive learning method naturally restricts how much the model can learn because of how its objective function is designed. To make contrastive learning effective, VPR needs to collect the most representative positive and negative examples for each query, which are the best and worst in-context examples. Consequently, a large amount of ranking observations remain unused during training. (2) Deciding on the Comparison Set: When considering the ranking metric, it is best to exhaustively compare each example against all the candidates available. However, directly ranking all alternatives globally is often not always possible. Instead, we have to aggregate rankings from partial predictions. Pair-wise ranking, like in VPR, ensures consistent ranking predictions by comparing each option's similarity to the query individually. But it cannot grasp the overall relationship between options. Conversely, methods with larger capacity, such as list-wise ranking, struggle to create a unified ranking due to inconsistencies among different partial predictions. Anyway, balancing feasibility and capacity is crucial for VICL sample selection algorithms.To solve these problems, this paper proposes a systematic VICL example selection framework, dubbed Partial2Global, towards the global optimal in-context examples selection. Partial2Global involves several list-wise partial rankers based on transformers to rank multiple alternative samples, and a consistency-aware ranking aggregator to achieve globally consistent ranking from the partial rankers.(1) To approximate the ground-truth ranking metric, we propose a transformer-based list-wise ranker to rank multiple alternative samples in a more informative and comprehensive manner. This model takes features of several alternatives together with the query sample from pretrained foundation models as input. The ranking predictions can then be inferred with more comprehensive knowledge from multiple alternatives and query sample by merging information among class tokens regarding each sample. Through meta-learning based training strategy, our model can learn a generalizable ranking metric for each specific task, thus better facilitating VICL example selection. (2) To approximate the global ranking, we propose a consistency-aware ranking aggregator to strike the balance between feasibility and ranking capacity. Particularly, partial and noisy ranking predictions from the trained ranking model are collected and divided into several groups, which can be reorganized and aggregated by learning a global ranker that has minimized average distance with all ranking groups. The resulted global ranking enjoys better ranking consistency and thus provides better in-context examples to boost VICL performance. Contributions of this work are as follows:• We emphasize the importance of the global consistent ranking relationships and correct ranking metric approximation in the selection of in-context samples for VICL. • We propose to approximate the optimal global ranking with a transformer based list-wise ranker, which can process more comprehensive information and produce stronger predictions. • We adopt a consistency-aware ranking aggregator to aggregate the partial predictions generated by our ranking model. • Extensive experiment results show that our method consistently works well in various visual tasks based on pretrained in-context learning model, showing the practical value of such a method. 
related works: 
 In-context learning. Given the current trend of scaling up model sizes, the large-scale models such as large language models (LLMs) [2] and their multi-modal counterparts [10] are shown to gradually learn the ability to perform in-context learning, i.e. inference with knowledge provided by few-shot context samples, rather than training the model with extra data. For example, Pan et. al. [15] designed an in-context learning system to generate symbolic representation, which can then be used for logical inference. Zhang et. al. [33] proposed to leverage in-context learning to edit factual knowledge in LLMs. Handel et. al. [7] showed that in-context learning works by compressing training set into task representations which can modulate models during inference for desired output. Such a property enables nonprofessional users to drive the model simply with several examples, the same as someone teachers his children. Recently, MAE-VQGAN [1] and Painter [24] show that by learning to inpaint the image grid composited by support and query pairs, the vision models can also learn the in-context learning ability. VPR [32] [20, 13, 27, 28, 26, 25] . In in-context learning, sample selection focuses on finding the best prompt example for a given query. This is typically done by training additional ranking models. These models compare a list of examples with the query based on their similarity in a specific property. Burges et al. [3] proposed an objective function for pairwise ranking, which was further extended to listwise ranking [23] . Recently several works focus on learning ranking with transformer structures. For example, Kumar et al. [9] proposed to adopt attention mechanism to interact between samples to be ranked. Liu et al. [11] proposed to build the ranking model based on GPT model. In this paper we propose to leverage the learning to rank technique to build a transformer-based list-wise ranker, which can learn more comprehensive information than previously used pair-wise ranker learned with contrastive learning.Aggregating partial ranking predictions. Jiang et. al. [8] proposed HodgeRank as a Hodgetheoretic approach to statistical ranking, which can provide global ranking with incomplete and inconsistent ranking data. The comprehensive insight of HodgeRank made it widely applicable to quality of experience assessment. For example, Xu et al. [30] adopts HodgeRank for subjective video quality assessment based on random graph models. The similar idea is also applied to crowdsourced pairwise ranking aggregation [31] . In this paper we take inspiration from Hodge theory to build a novel framework for visual in-context learning. While such a strategy has never been studied before, we show that our framework can significantly boost the visual in-context performance by selecting better examples. 
methodology: 
 Preliminary: Visual in-context learning. Visual In-Context Learning (VICL) aims to build the inference procedure of each testing sample based on knowledge provided by in-context examples. MAE-VQGAN [1] instantiates such an idea through image inpainting. Concretely, given a query sample, an example image is selected randomly along with its annotations (e.g., bounding boxes, masks, etc.) from training set. Then the example image, example label, query image and a random query label to be predicted are arranged as an image grid so that the query label is placed in the lower right corner. Then this composed image is used as input to an inpainting model. The masked patches are predicted by a transformer autoencoder and further decoded with the pretrained VQ-VAE [22] . To ensure the generalization ability, MAE-VQGAN is pretrained on a large-scale dataset collected from arxiv papers. While it shows promising results among different kinds of tasks, the important problem of in-context example selection is omitted, which leaves this method great potential for improvement.Preliminary: Visual Prompt Retrieval. Visual Prompt Retrieval (VPR) [32] mainly focuses on the in-context example selection based on MAE-VQGAN. Specifically, VPR starts with the heuristic that more similar an image is to the query image, the better it is as the in-context example. Based on this, the unsupervised and supervised variants of VPR are designed. In the unsupervised setting, VPR directly selects the most visually similar training images for each query image. As for the supervised setting, VPR first creates of a performance metric dataset collected from the training set, symbolized as {x i q , X i R , y i } N T r i=1 , where x q denotes query sample, X R denotes a set of K alternatives to be ranked, and y denotes the performance metric, e.g. IoU for segmentation or accuracy for classification, for pretrained MAE-VQGAN when testing x q on specific task using X R as in-context examples respectively. Then a metric learning model is trained to learn the in-context performance of training samples, i.e. the best performing samples are labeled as positive samples and learned against the worst performing ones as negative samples. Such a method, while outperforming the naive MAE-VQGAN, still cannot fully explore the knowledge regarding the ranking of in-context examples with the unprocessed partial observations. To this end, we propose a novel method to boost VICL with more proper in-context example selection.Overview. To boost VICL with better in-context example selection, we in this paper propose a novel pipeline including two main steps. First we train a ranking model (Sec. 3.1) based on the performance metric dataset collected in the same way as VPR. Different from VPR, we utilize an extra transformer model to learn list level ranking instead of instance level scoring so that inner relationship between alternatives can be better utilized for ranking. Once the ranking model is trained, the global ranking of alternative set needs to be gathered from partial ranking predictions. To this end, we propose to adopt a consistency-aware ranking aggregator (Sec. 3.2) to process predictions to provide globally consistent ranking results, thus better facilitating VICL.In order to learn a more proper ranking metric approximation, we propose to adopt a transformerbased list-wise ranker. Our methodology is rooted in the principles of VPR. Given the query sample x q and the alternative set X R , we first sample a k size subset x from X R . Then a ranking model ϕ k is built to provide ranking prediction of x as ϕ k (x, x q ). Concretely, features z q ∈ R (N +1)×C , z ∈ R k×(N +1)×C corresponding to x q , x are extracted with pretrained transformer model ϕ such as CLIP [17] or DINO [4] , where N + 1 features include class token and patch tokens, C denotes feature channels. Following the feature extraction, we concatenate all these features to form a feature sequence ẑ ∈ R (k+1)(N +1)×C . For the sake of simplicity and to facilitate further processing, we rearrange this sequence such that all class tokens are positioned at the beginning. This sequence ẑ is then processed through several newly introduced transformer layers. These layers are designed to enhance interaction between the global-level and local-level features contained in different images. This interaction is crucial as it allows the model to gather the necessary information for ranking the alternatives. Upon completion of these layers, we collect the class tokens of alternatives z cls = ẑ[1:k+1,:] . These tokens are then processed through linear layers to generate the ranking prediction ŷ ∈ R k . This prediction ŷ serves as an indicator of the ranking of the alternatives, providing a quantitative measure of their relevance to the query sample.Training objectives. Our ranking model is optimized with a composed objective as follow:L = k i,j=1,i̸ =j max(0, 1(y i > y j )(x i -x j + δ)) Lmargin + NeuralNDCG(τ )(ŷ, y) Lsort + MSE(ŷ ′ , y) Lreg (1)where L margin denotes the pair-wise marginal ranking loss, L sort denotes list-wise Neural-NDCG [16] , δ and τ represents the loss coefficients for two loss terms respectively. L margin and L reg act a similar role as the supervised contrastive loss in VPR, which encourages the model the predict higher score for better samples in each pair. L sort , on the other hand, can drive the model to learn inner relationship among more alternatives, thus leading to better ranking predictions.For this section, we will introduce how to obtain consistent global rank for the alternative set given the trained ranking model as introduced above.Motivation. In the process of selecting in-context examples, it is crucial to identify the most suitable candidates from the available pool. However, the ranking model we employ only provides partial ranking predictions on a subset instead of the full alternative set. A naive solution would be splitting the alternative set into non-overlapping subsets, which are then processed with the trained ranking model respectively. Then the top ranked samples from each subset are gathered together as a new alternative set, which is then repeatedly split and ranked again until there is only one sample.While such a method can provide reasonable results as we will show in the experiment, it can suffer from three main problems: (1) In total, K k subsets with size k can be randomly sampled from an alternative set with size K. However, the naive method only utilizes ⌈ K k ⌉ subsets, which is extremely Algorithm 1 Consistency-aware ranking aggregator Input: Train set X train , query sample x q , trained ranking models {ϕ k }, alternative set size K.1: Alternative set X R = topK x∈Xtrain (sim(x, x q )) 2: Initial preference matrix set S := ∅ 3: for rank-k model ϕ k do 4:Build observation pool X k from X R 5:for randomly shuffled X i k from X k do 6:R i k = x∈X i k ϕ k (x, x q ) 7: Aggregate S i from R i k 8: S = S S i 9:end for 10: end for 11: Aggregate global ranking r as Eq. 2) Since this naive method is iterative, the selection error would accumulate, leading to unbearably wrong prediction. (3) It is inefficient to fetch other top ranked samples such as second or third best ones with this method. Therefore a new method, which can directly provide global ranking prediction by fully utilizing information contained in the alternative set, can thus better serve the purpose of in-context example selection. To this end we propose the consistency-aware ranking aggregator, which allows us to derive a global ranking from the partial information, thereby making it possible to rank all candidates in relation to each other while guaranteeing global consistency.Algorithm. Our method starts with gathering enough sufficient information from all alternatives. Given the query sample x q , alternative set X R and a trained k-length ranking model ϕ k , we can first build an observation poolX k = {X i R } Np i=1, where X i R denotes a randomly shuffled variant of X R . Then for each X i R we follow the naive ranking method to split it into ⌈ K k ⌉ non-overlapped k-length sequences and rank these sequences with ϕ k , resulting in a predicting set R i k . The predictions are further aggregated into a preference matrix S i ∈ R K×K . If the m-th alternative is favored over the n-th alternative, we let S i mn = 1 otherwise S i mn = -1. If no predictions are related with m and n-th alternatives, then S i mn = 0. S i can then be transformed into an pair-wise indication setE i = {(m, n)|S i mn ̸ = 0}.In this way we can finally get a preference matrix set S = {S i }. Denoting the global ranking of X R as a score vector r ∈ R K , in which higher score denotes higher ranking. If we have the oracle ranking r * K , then if m-th candidate is favored than n-th candidate we can get r * m -r * n > 0. As a result, we can formulate the ranking problem as follows,EQUATIONTo get the solution we can reformulate Eq. 2 as a Least Square problem by introducing a transformation matrixD i ∈ R ⌈ K k ⌉×K, each row of which is a sparse vector with only the two indices from the pairwise set E i as 1 and -1. Then we have the following formulation,EQUATIONBy solving Eq. 3 we can directly get a proper global ranking r. Furthermore, multiple ranking models can be engaged in this process by extending S with preference matrices calculated from these different models, which can in turn enhance the ranking process with more comprehensive information. Compared with the naive ranking method, this method involves more partial observations from X R , while avoiding the accumulative error, thus being both effective and efficient.Analysis. It is straightforward to see the merits of the proposed method. First, compared with the naive strategy, solving Eq. 3 can directly lead to a complete ranking results, avoiding repetitive computation for more top ranked examples. Second, it is inevitable that the trained ranking model wrongly predicts the ranking of some alternatives. Under such circumstance, the naive strategy has now way to fix such an error. On the contrary, since a same alternative pair can result in different predictions by our list-wise ranker, the proposed algorithm as in Alg. 1 can take into account all contradicting predictions, thus leaving potential for rectifying the wrong predictions.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: Our research focuses on general visual tasks, which totally conforms the Ethics Code.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
dataset and setting: 
 Dataset. We follow MAE-VQGAN [1] and VPR [32] to adopt three tasks including foreground segmentation, single object detection and colorization. For foreground segmentation, Pascal-5 i [19] is utilized which contains 4 data splits. We conduct experiments and report the mean intersection over union (mIoU) on all splits together with the averaged mIoU among these splits. For single object detection, Pascal VOC 2012 [5] is used. Images and predictions are processed the same as in MAE-VQGAN, in which mIoU is adopted as metric. For colorization, we first sample 50000 training data from ILSVRC2012 [18] training set. Then a test set randomly sampled from the validation set of ILSVRC2012 is used to test the model with mean squared error (MSE) as metric.Implementation Detail. Considering the training data size for each task, we adopt different sequence length for ranking. Specifically, we train rank-5 and rank-10 models for foreground segmentation and single object detection, while rank-3 and rank-5 models are trained for colorization. The training data is built by first selecting 50 most similar images from the whole training set for each image as its alternative set. Then we randomly sample 20 sequences with the required length from this alternative set for training. For all experiments we adopt DINO v2 [14] as feature extracter. AdamW optimizer [12] is used with learning rate set as 5e-5 and batch size set as 64. We utilize 4 V100 gpus to cover all experiments.Competitors. We choose three methods as our competitors, including MAE-VQGAN, VPR which contains UnsupPR and SupPR and prompt-SelF [21] , which mainly facilitates an ensemble strategy.To fairly compare our methods with these previous ones, we report main results of our models both with and without the test-time ensemble.The experiment results are shown in Tab. 1. Note that we omit the result of our model with voting strategy for colorization since prompt-SelF does not report this term and there is no fair competitors in this setting.As in Tab. 1, for all three tasks, our model receives best performance on both variant. Specifically, when not using voting strategy, our model outperforms the strongest competitors SupPR by 2.84 in terms of average segmentation mIoU, while the superiority is consistent on other two tasks. When using voting strategy, our model receives about 4 higher mIoU for segmentation and 2 higher mIoU for detection, leading to better results than prompt-SelF. One would note that while SupPR was designed to learn a better metric than the naive visual similarity used for UnsupPR, its performance on colorization is exactly the same as UnsupPR. On contrary to that, our model makes a huge leap with 0.04 less MSE, which is not only much better than both UnsupPR and SupPR but also doubles the improvement these two methods have against the random strategy in MAE-VQGAN. This can prove the effectiveness of our proposed ranking model for selecting in-context examples.To further illustrate the advantage of Partial2Global, we visualize several samples for segmentation and detection in Fig. 3 and Fig. 5 . We find that in some cases the in-context examples selected by VPR can let MAE-VQGAN generate totally wrong results, especially when multiple objects are presented in the query. For example, in the first sample of Fig. 3 , the query image contains two monitors, while the target label is only related to the left one. While both VPR and our method select images of monitors as in-context example, VPR results in 0 IoU while prediction using our example is much better. This may be resulted from the spatial relation of the object of interest in the example images. In the example selected by VPR, the monitor is placed at the right, which is the same as the non-target monitor in the query image, thus leading to wrong guidance and prediction. Apart from that, VPR tends to select small objects for detection, as shown in Fig. 5 . This can lead to failed detection even though the query objects are large enough for huamns to detect them. In comparison, our method can generally select more proper in-context examples, thus enjoying better performance. The IoU is listed below each image grid.To fully validate the effectiveness of each module in our method, we conduct a series of ablation studies. For all ablation studies, we do not use voting strategy. The experiments contain both foreground segmentation and single object detection if not specified.Effectiveness of different modules. First of all we provide comparison among variants with different designs. Specifically, we consider three models: (1) Rank-10 Naive: A rank-10 model is trained, of which the ranking predictions are directly used for example selection. (2) Rank-10 Aggr.: After training the rank-10 model and getting the ranking predictions, we use the proposed consistencyaware ranking aggregator described in Sec. 3.2 to process these data, then we pick the top-ranked sample in the post-processed global ranking as in-context example. (3) Rank-{5,10} Aggr.: Our full model, in which the ranking prediction of both ranking models are mixed and processed with consistency-aware ranking aggregator to get the final global ranking. The results are presented in Tab. 2. We find that the naive ranking model enjoys 1.54 higher average mIoU than SupPR, which supports our motivation of using ranking model instead of pair-wise contrastive learning model. Since ranking model can make better use of the training data, and comparing multiple samples enables the model to discover the inner relationship between them, rather than simply judging if one sample is better than another one. Moreover, We find that using consistency-aware ranking aggregator to process the ranking predictions can lead to further improvement, which can be enlarged by including results from rank-5 model. This is because the predictions of ranking models can be thought of as from different annotators. Even for a single model, its predictions will not be consistent when given two sequences which share partial same samples. Therefore, simply utilizing the inconsistent predictions can lead to suboptimal results, and consistency-aware ranking aggregator helps regulate the predictions to produce more globally consistent ranking results, thus having better results.A basic claim in VPR is that the more similar an image is to the query image, the better it is as an in-context example. Nonetheless, VPR utilizes performance metrics instead of visual similarity for contrastive learning and provides better results, which makes one wonder if such a claim is indeed solid. To check this out, we take a look at the best performing in-context examples selected by our method. The results are shown in Fig. 4 (a) and (b). We first visualize the correlation between visual similarity, which is computed by extracting CLIP features for each image and then calculating and averaging the cosine similarity between the alternatives and query, and mIoU on segmentation for both VPR and our method. It is clear that visual similarity can be a basic heuristic for choosing visual in-context examples. Generally well performing examples appear to share high similarity with query samples. On the other hand, there are also large amount of failure cases with high similarity, which indicates the ground principle should be much more complex than visual similarity. Based on this conclusion, we further visualize several cases in Fig. 4 (c) selected by our method, which better shows that one should consider more than visual similarity and stress other factors such as object size, spatial position. On the opposite, we find that the background similarity, which can contribute significantly to the visual similarity, is hardly related to the quality of in-context examples. Robustness among different backbones. We compare our model with two variants with other pretrained feature extractor: CLIP ViT-L/14 and DINO v1. The results are shown in Tab. 3. First, directly using ranking prediction from our ranking models through the naive ranking strategy performs better than VPR no matter which backbone is utilized. VPR also conducted such an experiment to try their method with different backbones such as CLIP, EVA and supervised ViT, but none of those results can outperform ours, which illustrates the efficacy of our design of ranking model against the contrastive learning based model. Second, adopting consistency-aware ranking aggregator consistently improves the naive ranking method by nearly 1-2 mIoU for both segmentation and detection, which is compatible with the results in Tab. 2, showing the robustness of the proposed method among different backbone choices. Third, the in-context performance is not totally correlated with the capacity of backbones. It is commonly known that DINO v2, as an improved version of DINO v1 with more comprehensive objective functions and more data, should be equipped with higher capacity. However, we find that DINO v2 performs worse than DINO v1 on segmentation whether naive ranking or consistency-aware ranking aggregator is adopted. We think such results can be attributed to the different learning difficulty when using these backbones, and it would be interesting to have further research on the impact of backbones for visual in-context learning.Consistent optimality of ranking results. Apart from the strong performance of our method, we further find that the proposed consistency-aware ranking aggregator can interestingly introduce consistent optimality to the ranking results. That is to say, apart from the top ranked sample, the second best and third best samples should also be generally better than other samples in the alternative set as in-context example. To show that we test two strategies: (1) directly using samples ranked second and third as in-context examples and (2) adopting a simple late fusion method to average predictions generated with 2, 3, 5 top ranked samples. The results are shown in Tab. 4. One can find that using top-3 best ranked sample have comparable results. For segmentation, the #2 rank samples only have 0.21 lower mIoU than #1 rank samples, and #3 rank samples are even slightly better than #2 rank samples, which evidences the consistent optimality. The results are consistent for detection. Moreover, fusing 2 or 3 samples together leads to 0.73 and 1.43 higher average mIoU, which also supports the consistent optimality brought by the consistency-aware ranking aggregator. Such improvement saturates when using 5 samples, indicating that samples ranked 5-th or later are significantly worse and cannot provide more useful information for the in-context inference. 
conclusion and discussion: 
 This paper proposes a novel pipeline for in-context example selection in Visual In-Context Learning (VICL). Specifically, we design a transformer-based ranking model, which can provide desirable ranking predictions by well utilizing the inner relationship among alternatives. The ranking results are further aggregated by the proposed consistent-aware ranking aggregator, which achieves desirable global ranking prediction by transforming the problem into a least square problem. Our method receives state-of-the-art performance on three different tasks including foreground segmentation, single object detection and image colorization, showing great potential of improving VICL researches.Limitations. While our proposed method can provide good in-context examples, the basic performance still highly relies on the quality of the in-context learning model. The MAE-VQGAN used in this paper is rather an early stage trial for visual in-context learning, with many limitations in terms of its structures and applications. We believe that our model can further help other stronger visual in-context learning methods, leading to better in-context performance.Broader impacts. Our work will not lead to significant negative social impacts. One main problem is that if the data used for in-context learning contains biases, the models trained on this data may also exhibit these biases, leading to unfair or discriminatory outcomes. This is particularly concerning in areas like facial recognition or predictive policing, where biased models could disproportionately affect certain groups. Solving such problems would be a large future research topic. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please refer to the Sec. 5 in the main paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We have not released our model weight yet.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite all used public datasets and pre-trained models in Sec. 4.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: NeuralNDCG. The list-wise ranking loss NeuralNDCG adopted in this paper is a differentiable approximation of Normalised Discounted Cumulative Gain (NDCG). We simply quote the definition here from [16] for better understanding. Specifically, for x denoting a sample and y denoting its score, NDCG can be calculated asEQUATIONwhere π f denotes the predicted ranking, π * is the ground truth ranking regarding score y. Neural-NDCG works by substituting the discontinuous sorting operator with NeuralSort, which results in an approximated permutation matrixEQUATION)where A x [i, j] = |s i -s j |, 1 denotes a vector filled with value 1, τ is a temperature parameter. Then the NeuralNDCG can be calculated asEQUATIONN euralN DCG(τEQUATIONwhere scale(•) is Sinkhorn scaling. Transferability of Partial2Global. To test the transferability of the prompt selection method and further reveal the potential of our method, we can add the demonstration suggested by conducting the following experiment for both SupPR and our method: we use models trained on each fold of segmentation and apply it to other folds. The results are shown in Tab. 6. We find that while both two methods degrade in the transfer learning setting, our method still outperforms SupPR in general. This interesting results indicate that, the training data size of each fold is insuifficient for training a robust and generalizable ranking model. On the other hand, this again indicates that prompt selection for VICL cannot be simply based on visual similarity, as claimed in our paper. Efficiency of our method. One would ask if the proposed method would suffer from poor efficiency, compared with SupPR. In general, the usage of list-wise ranker and the ranking aggregation will inevitably introduce additional computational cost, while the increased complexity during inference is affordable under common circumstances. Specifically, we provide the training and inference time cost as follows. (1) The training of list-wise ranker on the colorization task, which contains about 500000 ranking sequences, takes about 10 hours on 8 V100s. Once the model is trained it can be directly used for any other queries with the same ranking criteria as the training task without any further finetuning. ( 2) During inference on one V100 gpu, our proposed pipeline requires about 1.17s to rank 50 alternatives for each query sample in the complete process, including feature extracting (0.3s), sub-sequence ranking with list-wise ranker (0.8s), and ranking aggregation (0.07s). Note that some techniques could be utilized to accelerate this process. For example, when we prepare the extracted features in advance (which is reasonable given the candidate set can be prepared in advance), we can skip the feature extraction stage and reduce the time cost by 0.3s. With engineering works, the inference time cost can be further reduced. The detailed inference speed (feature extraction included) given different alternative set size is presented as in Tab. 7. Upper bound of selecting different in-context prompts. Another interesting problem is to examine the 'upper bound' of performance by adopting different in-context prompts. To further provide insight to this task, we try to examine an upper bound: directly testing all alternatives for each query in the segmentation task and presenting the best IoU in Tab. 8. While our proposed method is much better than SupPR, it still leaves great potential for better performance, which we will take as future works. Sensitivity test for hyper-parameters. In general, our method is robust against changes of hyperparameters. To show this, we try the suggestions to conduct ablation studies on these two hyperparameters, whose results are presented in the following table. For delta which denotes the margin, using 0 margin leads to worse results while larger margins would be better for learning ranking models. For tau which is the temperature coefficient in NeuralNDCG, we simply use the best setting tau=1 from the original paper. As can be seen in Tab. 9, using smaller temperature would not lead to better results. Nonetheless, all hyperparameter settings enjoy better performance than SupPR, validating the effectiveness of our method. Impact of alternative set size on performance. Our choice of alternative set is the same as VPR, i.e. selecting 50 most visually similar samples based on CLIP features. Typically, this choice is good enough as shown in previous works like VPR. Here, to further investigate the impact of data quality, we thus tried the suggestion to test our method on all folds of segmentation task with 25 or 100 alternatives for each query. The results are shown in Tab. 10. One can find that our method is also very robust against different sizes of alternative set. Effectiveness of different terms in our proposed loss. We have conducted an additional ablation study to compare models trained for colorization without each loss term. The results are shown in Tab. 11. In general all three loss terms contribute the final performance, with L sort plays the most important role. Justification: We have analyzed the limitation of our main in Sec. 5. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The paper received two Weak Accept and two Borderline Accept responses. The AC read the paper, rebuttal, and discussions, and agree with the reviewers' recommendations. It is also encouraged that the authors incorporate reviewers' comments in their final version. Congratulations to the acceptance!

Reviews: review: paper_summary: paper_summary: This paper studies the demonstration retrieval mechanism for visual in-context learning. The authors propose Partial2Global which uses a transformer-based ranker and a consistency-aware aggregator to find the optimal demonstration. Experiments show that Partial2Global outperforms existing methods in tasks like segmentation and object detection, setting new state-of-the-art results.
strength_weakness: strength_weakness: 1. Since training exclusively on a single dataset is limiting, I would appreciate a demonstration of transfer learning performance to prove its adaptability and universality across a broad range of data.

2. It would be beneficial to analyze the efficiency of your model to the size of the retrieval set, as complexity is a crucial aspect.

3. It would be insightful to examine the "upper bound" of performance, such as by concatenating each demonstration with a query to see what the maximum Intersection over Union (IoU) is, and to assess how much room there is for improvement from this upper bound.
questions: questions: See the Weaknesses.
limitations: limitations: N/A.
paper_summary: paper_summary: This paper addresses Visual In-Context Learning (VICL), which uses examples to help models learn new tasks. The main challenge is choosing the best prompt to improve learning and prediction. The authors introduce Partial2Global, a new method to find the best examples for each query. This method uses a transformer-based ranker to compare alternatives and a ranking aggregator for consistency. Experiments show that Partial2Global outperforms other methods in tasks like segmentation, object detection, and colorization, setting new performance standards.
strength_weakness: strength_weakness: 1.The paper does not include ablation studies on the critical parameters, such as δ and τ. These studies are essential for understanding how variations in these parameters affect the model's performance. Without this information, it is difficult to assess the robustness and sensitivity of the model to changes in these parameters.

2.The paper introduces heuristic methods for ranking in-context examples using a list-wise ranker and a consistency-aware ranking aggregator. However, there is no analysis provided on the computational complexity and time cost associated with these methods. Given the potential for high computational demands, especially with large datasets, it is important to understand the scalability and practicality of these approaches.

3.Figure 4 of the paper presents results related to visual similarity, but it lacks a clear explanation of how this similarity is computed. The method or formula used to calculate visual similarity is not provided, which is crucial for the reproducibility of the results and for understanding the underlying methodology.
questions: questions: Please see Weaknesses.
limitations: limitations: The authors address the limitations and potential negative societal impacts of their work.
paper_summary: paper_summary: This paper addresses the fundamental problem in Visual In-Context Learning (VICL), which is how to select the best prompt. Specifically, it focuses on the ranking problem. The paper proposes an algorithm called Partial2Global, which conducts an in-context example selection framework to find the global optimal prompt. The effectiveness of the proposed algorithm is demonstrated through several tasks, such as segmentation, object detection, and colorization.
strength_weakness: strength_weakness: The method seems to incur additional costs due to the ranking process. Is there any further analysis on this?
questions: questions: Please see the weakness part.
limitations: limitations: Please see the weakness part.
paper_summary: paper_summary: The paper proposes a framework called Partial2Global for ranking in-context examples in visual language models. This method uses a transformer-based list-wise ranker and a consistency aware ranking aggregator to approximate the global optimal prompt selection. The framework is validated through experiments on tasks such as foreground segmentation, single object detection, and image colorization, demonstrating its effectiveness over existing methods.
strength_weakness: strength_weakness: 1.The workload is relatively low. I think this work should also explore the impact of models with a larger number of shots on ICL. Additionally, it should investigate the impact of data quality in the alternative dataset, and include the importance of different parts of the loss in the ranking model during ablation experiments in formula (1).
2.The conclusions presented in some charts are not intuitive. For example, in Figure 4, can the similarity of our method and the visual similarity of VPR be displayed in the same chart?
3.Compared to previous methods, although it achieves a much better level, the improvement is relatively small.
questions: questions: See weakness part
limitations: limitations: NA

Comments: title: Thanks for the response. I maintain my positive score.
comment: Most of my concerns are solved, Thanks for the response.
I appreciate the author's response. I have read the author's responses, and most of my concerns have been addressed. Therefore, I have decided to maintain a 'Borderline Accept' stance on this paper.
I appreciate the authors' response. I have read the response and some of my concerns (Q1 & Q3) have been addressed. I decide to keep a Weak Accept for the paper.