Towards Reliable and Holistic Visual In-Context Learning Prompt Selection
Authors: Wenxiao Wu
Jing-Hao Xue
Chengming Xu
Chen Liu
Xinwei Sun
Changxin Gao
Nong Sang
Yanwei Fu
Venue: NeurIPS 2025 poster ()
Keywords: Visual In-Context Learning
Jackknife Conformal Prediction
Covering Design
Primary area: deep_learning
TL;DR: Our proposed RH-Partial2Global adopts a jackknife conformal prediction-guided selection strategy to construct reliable alternative sets and a covering design-based sampling strategy to ensure holistic coverage of local pairwise preferences.

Abstract: Visual In-Context Learning (VICL) has emerged as a prominent approach for adapting visual foundation models to novel tasks, by effectively exploiting contextual information embedded in in-context examples, which can be formulated as a global ranking problem of potential candidates. Current VICL methods, such as Partial2Global and VPR, are grounded in the similarity-priority assumption that images more visually similar to a query image serve as better in-context examples. This foundational assumption, while intuitive, lacks sufficient justification for its efficacy in selecting optimal in-context examples. Furthermore, Partial2Global constructs its global ranking from a series of randomly sampled pairwise preference predictions. Such a reliance on random sampling can lead to incomplete coverage and redundant samplings of comparisons, thus further adversely impacting the final global ranking. To address these issues, this paper introduces an enhanced variant of Partial2Global designed for reliable and holistic selection of in-context examples in VICL. Our proposed method, dubbed RH-Partial2Global, leverages a jackknife conformal prediction-guided strategy to construct reliable alternative sets and a covering design-based sampling approach to ensure comprehensive and uniform coverage of pairwise preferences. Extensive experiments demonstrate that RH-Partial2Global achieves excellent performance and outperforms Partial2Global across diverse visual tasks.

Towards Reliable and Holistic Visual In-Context Learning Prompt Selection
Wenxiao Wul
Jing-Hao Xue
Chengming Xu
Chen Liu
Xinwei Sun
Changxin Gao
Nong Sang' Yanwei Fu
Huazhong University of Science and Technology Shanghai Innovation Institute
University College London Tencent Youtu Lab
The Hong Kong University of Science and Technology Fudan University
fwenxiaowu, cgao, nsang) @hust. edu. en jinghao xue@ucl. uk chengmingxu@rencent com cliudhêconnect. ust.hk {sunxinwei, yanweifu @fudan. edu. cn
Abstract
Visual In-Context Learning (VICL) has emerged as a prominent approach for adapting visual foundation models to novel tasks, by effectively exploiting con- textual information embedded in-context examples, which can be formulated as global ranking problem of potential candidates. Current VICL methods, such Partial2Global and VPR, are grounded in the images more visually similar to query image serve better in-context examples. This foundational assumption, while intuitive, lacks sufficient justification for its efficacy in selecting optimal in-context examples. Furthermore, Partial2Glooaa constructs its global ranking from a series of randomly sampled pairwise pref- erence predictions. Such reliance random sampling lead to incomplete coverage and redundant samplings of comparisons, thus further adversely impact- ing the final global ranking. To address these issues, this paper introduces enhanced variant of Partial2Global designed for reliable and holistic selection of in-context examples in VICL. Our proposed method, dubbed RH-Paalial2Global, leverages a jackknife conformal alternative sets and a covering design- based sampling approach ensure com- prehensive and uniform coverage of pairwise preferences. Extensive experiments demonstrate that QH-Partial2Gloobll achieves excellent performance and outper- forms Partia/2Global across diverse visual tasks. The source code is available in https://githu..com// Wu- Wentaao/tH--aaataal2iiiaaa
assumption that
ierectioonnuiiddd strategy to construct reliable
Introduction
Inspired by the success of in-context learning in natural language processing, visual in-context learn- ing (VICL) was born on demand as a promising paradigm for leveraging Visual Foundation Models (VFMs) in vision tasks. By conditioning VFMs few in-context examples, typically image-label pairs, VICL facilitates efficient adaptation to various downstream tasks, such as segmentation and detection 2 image editing [3], cross-modal content reasoning 4 well as low-level tasks
key challenge in VICL lies in selecting the most suitable in-context examples corresponding to given query image to achieve optimal performance. Given that a set of candidate examples is typically available, recent works [2) 6 commonly formulate VICL a global ranking problem. Partial2Global [6]. state-of-the-aa method, tackles this by first training list-wise partial ranker through meta-learning This ranker is applied randomly partitioned candidate sequences for par- tial rankings, from which consistency aggregator subsequently infers global ranking.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

While Partial2Global marks a significant step forward for VICL, two critical limitations remain unresolved: (1) The similarity-piority assumption. Both Partial2Global and VPR rely on a heuristic known as the iimilarty-prrioiig assumption that the more visually similar candidate image is to the query, the better it serves as an in-context example. Although Partia/2Global questions the reliability of this assumption, it lacks a statistical foundation to support this critique and does not offer concrete alternatives. As a result, the assumption remains unchallenged in a rigorous or systematic way, and its influence on performance is left largely unexplored. (2) The strategy of random sampling. In constructing a global ranking from partial rankings, it is ideal to uniformly and exhaustively cover all pairwise preferences among candidates. However, the reliance of Partial2Global random shuffle operations for partial predictions often fails to capture all inter-candidee introduce redundant comparisons while neglecting informative ones, ultimately degrading ranking accuracy. Thus, a more principled sampling strategy is needed to achieve comprehensive coverage of candidate relationships while simultaneously minimizing redundancy and iteration overhead.
relationships and may
Towards Reliable and Holistic VICL Prompt Selection, we build upon the Partial2Global frame- work and introduce an enhanced approach named RH-PartialZGloba. To address the two key limitations discussed earlier, RH-Partial2Glohal incorporates the following major improvements: (1) Reliable Selection Strategy via Conformal Prediction. To construct more trustworthy alterna- tive set of in-context examples, we propose a selection strategy based on conformal prediction. This method operates in a jackknife manner: for each training sample, it computes a consistency score that quantifies the alignment between the sample's 's quality when serving as an in-context prompt for other instances and its visual similarity to these prompted instances. By applying a threshold derived from a quantile of these scores at predefined confidence level, we identify subset of candidates exhibiting high reliability. This identified subset then used to filter the initial similarity-drie driven candidate pool, resulting in more robust and accurate selection of in-context examples. (2) Holis- tic and Efficient Sampling Strutegy via Covering Design. To ensure uniform and comprehensive coverage of pairwise candidate preferences, we integrate covering design into the local sampling process. This is operationalized by sequentially sampling from a randomly shuffled alternative set, guided by precomputed optimal covering design. Please note that the principles of covering design inherently ensure exhaustive coverage of preference relationships, while its property of minimizing sampling iterations contributes to the uniformity of pairwise preference sampling.
We summarize our contributions as four-folds:
We turn our attention necessary yet often neglected heuristic in VICL that images more visu- ally similar to a query image serve as better in-context examples. For the first time, we provide statistical evidence demonstrating that this similarity-prioriy assumption is not sufficiently robust.
We propose jackknife conformal prediction-based example selection strategy to identify and preserve reliable samples in the alternative set, thereby reducing ranking complexity and yielding more accurate predictions.
We incorporate covering design- design-based sampling strategy within the consistency ranking aggregator of Partial2Global, ensuring more holistic coverage of pairwise preferences and conse- quently leading to a more accurate global ranking.
Extensive experiments substantiate that our proposed RH- PGltaallallGoloall can significantly surpass the state-of-the-art Partial2Global framework across diverse vision tasks.
2 Related work
In-Context Learning. The recent expansion in the scale of large-scale models has led to remark- able advances in their ability to perform in-context learning, a process where models adapt to new tasks by conditioning on a small set of examples rather than undergoing further training. Notably, both large language models (LLMs) 7 and their multi-modal successors (8) have demonstrated
this capability. For instance, Pan et al. (9] leveraged in-context learning to construct symbolic rep- resentations that facilitate logical reasoning, while Zhang et al. [10) applied similar techniques to update factual content within LLMs. Parallel developments have emerged in the field of computer vision. Approaches such as MAE-VQGAN I and Painter (5) have shown that vision models can be trained to perform in-context learning by reconstructing masked regions in image grids composed of both support and query images. Building on this, VPR (2) addressed the challenge of selecting optimal in-context samples, introducing metrie network based on contrastive learning and perfor- mance evaluation. Prompt-SelF 11 extended this line of work by incorporating both fine-grained

---

and poarse-graines visual similarities, and proposed an ensemble approach that aggregates predic- tions from multiple permutations of in-context grids at inference time. Partial2Global [6] proposed a list-wise ranker and globally consistent ranking aggregator aiming for global optimal in-context prompts. Our work aims to find better in-context example, but different from previous works, we break the assumption that similarity between images can safely lead to better in-context examples. Other than that, we focus on leveraging conformal prediction and covering design to build reliable and holistic in-context example selection process.
13 14] is a distribution- and model- agnostic methodology that generates prediction sets with theoretically guaranteed coverage proba- bilities. Unlike traditional point predictions, CP quantifies uncertainty by producing a set of plau- sible outcomes rather than a single value. This ensures that the true outcome (e.g., label, value, or element) is contained within the prediction set with a user-specified confidence level. Conformal prediction has garnered significant attention in various tasks that require rigorous uncertainty es- timation, such multi-class prediction [15). benchmarking of LLMs [16), and robotic trajectory prediction [77]. CP methodologies can be broadly categorized into several types: full CP, split CP (or inductive CP) [88), transductive approaches like jackknife CP
Conformal Prediction. Conformal prediction (CP) [12
[99 20 and CP with cross- validation 21 22), and conformal risk control 23 (4]. Among these variants, jackknife CP is known for its tendency towards conservative predictions, rendering it partieularly suitable for appli- cations demanding high reliability and robustness. In this paper, we leverage jackknife CP to refine the construction of alternative sets, aiming retain highly reliable samples and thereby alleviate the difficulty of subsequent ranking tasks.
Covering Design. A covering design (25] is a fundamental combinatorial structure that addresses the problem of systematically covering all possible subsets of fixed size within a larger set. For- mally, a (K K, k I) covering design is a collection of k-element subsets (called blocks) from a K- element set, such that every /-element subset of the K-set is contained in at least one block. A primary objective in covering designs is to find the minimum number of blocks, C(K, k, t), sat- isfying the covering condition, highlighting their efficiency in ensuring comprehensive interaction coverage. In our work, motivated by the limitations of random sampling and the inherent need for complete and uniform coverage of pairwise preferences (i.e., f 2) among candidate examples, we replace the original random sampling with a covering design-based strategy, thereby facilitating a more systematic, comprehensive, and balanced sampling of these pairwise relationships.
3 Methodology
Preliminary of Partial2Global: Adhering to the similarity-priorins heuristic prevalent in methods like Visual Prompt Retrieval (VPR) 2 that images more visually similar to a query Tig serve as better in-context examples, Partial2Global (6) constructs an alternative set Yi of size K from train- ing set Xer = {-t M+1 for each query sample Ty in test set Mestt As for the training stage, Partial2Global proposes transformer-based list-wise ranker Φv of length k for local ranking and trains in a meta-learning manner. When it comes to the evaluation stage, Partial2Global system- atically infers global rankings from local predictions. This process begins by first constructing an observation pool comprising N, randomly shuffled variants of the alternative set Vg. For each such permuted variant, the K candidates are divided into into [ non-overlapping sub-sequences of length k. Each sub-sequence then ranked using Φ* and the resulting pairwise preferences are aggregated into a preference vector S which entries encode dominance relationships -1, or 0). Each vector S" is then converted into a pairwise indication set E², which explicitly lists all candidate pairs with non-neutral preference relationships derived from the ranker's predictions. The collection of these indication sets and preference vectors forms the basis for deriving a global ranking score vector r. The derivation of is then reformulated as a least squares problem using transformation N₂ matrix D'. which encodes pairwise comparisons from E', leading to min, Σe 2 1'1 Compared to naive ranking, this aggregation approach enhances both effectiveness through prefer- ence modeling and efficiency by mitigating sequential dependencies.
i==1
Overview. To advance VICL via superior in-context prompt selection, we propose two correspond- ing enhancements that address two key limitations in Partial2Global. First, to mitigate the sub- optimal criteria used for constructing initial alternative sets, we rely on the theoretical foundation of jackknife conformal prediction (Sec. 3.T to identify a prompt set consisting of high-confidence, reliable candidates. With this reliable set, refine the initial alternative set retaining only those
3

---

Table 1: Experimental results of Spearman's rank correlation test for each fold. The number and proportion of samples with statistically significant monotonic associations (p < 0.05) and the aver- age correlation (p) are reported.
Fold-0
Fold-1
Fold-2
Fold-3
1786/2279
2906/3309
4152/5030
1584/1986
#(p 0.05)
(78.37%)
(87.82%)
(82.54%)
(79.75%)
p
0.0548
0.0315
0.0345
0.0500
candidates present in their intersection, thereby the reliability of the inputs to the global ranking process. Once this high-confidence alternative set is established, local rankings are typically gen- erated through random shuffling of candidates prior aggregation. However, this process can lead to incomplete and redundant coverage of all pairwise relationships, potentially resulting in perfor- mance instability. To this end, we further introduce the use of covering designs (Sec. 3.2) to guide more structured and comprehensive sampling strategy, ultimately leading to more stable and accu- rate global ranking performance.
3.1 Conformal prediction-guides strategy for reliable candidate selection
This section details the construction of a reliable alternative set Va with user-specified confidence level a for query images, guided by the theory of conformal prediction with jackknife.
Motivation. To examine the validity of the similarity-priorie assumption, which posits that im- ages more visually similar to a query image serve as better in-context examples, we conduct an experimental hypothesis test based on the training set Xrm from Pascal-5' (26) dataset for the seg- mentation task. For each sample x; in Xir serving as the query, the remaining examples Xirn are used potential in-context prompts. We then derive two sequences for these potential examples: (1) their IoU scores when used as prompts for F;, and (2) their visual similarity scores to Tg. To assess the association between the two sequences,
calculate the Spearman's 's rank correlation coefficient E [-1, 1] and employ the Spearman's 's rank correlation test to obtain the p-value p E [0, 1]. Here, high Spearman correlation coefficient p indicates observations have similar rankings in terms of both their loU scores and their visual similarities, while a low suggests dissimilar rankings. Since the null hypothesis Ho of the Spearman's rank correlation test posits no monotonic association be- tween the two variables, a p-value below the significance level (e.g., 0.05) indicates rejection of Ho. thereby suggesting a statistically significant monotonic relationship. We conduct this procedure across all folds of Xi calculating both the average correlation ğ and the number and proportion of query samples for which p < 0.05. The experimental results are summarized in Table [
Analysis. The experimental results presented in the second row of Table | consistently indicate across all folds that there indeed exists statistically significant monotonic relationship between the quality of in-context examples (e.g.. IoU in segmentation tasks) and visual similarities. However, the average Spearman correlation coefficients á presented in the third row of Table D are notably low. This suggests that while a general statistical association is frequently present, the strength of this monotonic relationship is often weak. These observations underscore the need for a more robust criterion than pure similarity alone. Such a criterion should selectively identify and retain genuinely reliable candidate examples from an initial alternative set V/, which was broadly guided by the similarity- assumption, thereby refining the selection process. Hence our algorithm as follows.
Algorithm. We begin by directly using the original training set X [a:(r))) M ing set" for conformal prediction. Please note that we do not need Xern to train predictive model, because our conformity function f(-) will be just a predefined metric (e.g., the negative KI. Diver- gence or the Spearman correlation) that quantifics the consistency between the quality of in-context examples Q and their visual similarities S, as detailed below from Eq. to (3)
Ji=l as the "train-
Let FF be a pretrained inpainting model. Mathematically, when a sample 2.-7 used as an in- context prompt, the VICL process for any query results in an output F(,, 22. Given a function q. which typically evaluates the performance of F(., with respect to x the quality of this VICL process can be denoted as q(F(,, "ef"") ";"""") Consequently, for each otr E Xerns we assess its quality as a prompt by applying it as the prompt to all other samples 25 in Xer \ \ This yields set of M quality scores:
Q(xtT") (q(F(x))) " ,,"" x (r M+1
(1)

---

Algorithm Jackknife Conformal Prediction -oriiionn--ueddddddd Candidate Selection
Require:
Constructed training set Xtrn query sample zg. alternative set size K, pretrained inpaint- ing model F. conformity function f. confidence level or
1: Initial the conformity score set V := {-xc}
2: for -!!" in Eern do
3: Initial quality set Q(x:"")== := 0 and similarity set S(xtt") := D
4: for srfr in Xtrn ...T do
5: Compute the quality score as as Eq. q (F())) , ),
6: Update quality set: 2(x,"*) (a)))) U QJF ((F(atm F setrm) 2, slt
7: Compute the similarity score as Eq. ( s(af aftr)
8: Update similarity set: S(x)"*) S (()"")) Us(a)T ; aatr)
9:
end for
10: Compute the conformity score as Eq.03 3 (((tm) = 1(2(2)m)) s(a!")
11:
Update conformity score set: V = V (x)t"))
12: Compute the quantile qı-a (V) as 55
13:
Construct the reliable set as 6 : Va
{ag t E Xern (((ttT) > (V))
end for
15: Initialize alternative set 3/g for zg: D/g iop-Kaex... s (
16: Obtain the refined alternative set 3 as (q. 7): 2/g = Vo n2/
17:
return Dg.
Similarly, we define s(., =44")) as function measuring the visual similarity between the prompt at and any other query. For each xfrm, we compute its similarities to all other samples in Xyr ttrn forming set of M similarity scores:
S(x)")) = {s(a)"m "a"a""""""""" )) M+1 j=
(2)
Based on the above definitions, the conformity score {(x!r)) for each x!C/m is calculated by applying function f (c.g.. the negative KL Divergence or the Spearman correlation) its corresponding quality and similarity sets:
((--T)- = (i00)=((((( ".(a""
(3)
This procedure is repeated for every x. Xerns treating each in turn as the prompt whose confor- mity score is being computed, to constitute a jackknife procedure. Then, a set including all M + conformity scores can be denoted by
V (f(x;" M U--oo).
(4)
Subsequently, the corresponding determined from the empirical distribution of conformity scores in V:
qu-attielleeeeeeeeeeeeeeeooeeee Q1-a, which serves as the reliability threshold, is
Q1 a1-a( (V) the [(1 (M 2)]-th smallest E.
(5)
Crucially, this quantile qı qı (V) derived solely from the training set Xer and is thus independent of any specific query I.g from the test set or its initial candidate set. Thus, we can define global set of highly reliable candidate prompts Va derived from Xyrn:
Va = {.: E Xern E(x:"T) > (V)}
(6)
This set V. comprises samples whose conformity scores are in the top percentiles, indicating high reliability. We now describe how this globally reliable set V. is used to refine a query-specific alternative set y/, For given query Zq. we first construct the initial alternative set Yi based on the momilarity-priorii assumption: )/8 = ttop- s(xgg;
3). The selection of candidates in 3, can be seen as a procedure of conformal prediction test. Our core idea to test whether one element x in D,, can also satisfy {(x;3)= > Qi (V), i.e., ..rr E D.p The entire process can be viewed as filtering a known D,, to obtain the refined set Y using Va. which can be expressed as
y = V., n y, = {x} : E(xx)) > ()), E V,)- In the procedure for constructing V.., our primary focus is on evaluating the intrinsic reli- ability of each 2 as a potential prompt, independent of any specific test query Ty. Consequently, while no guaranteed optimality for any individual test query zig this strategy aims to ensure that the selected reliable prompts offer overall satisfactory performance by virtue of their proven reliability on X The whole procedure is outlined in Algorithm
Remark.
5

---

3.2 Covering design-basen based strategy for holistic sampling
In this section, we elaborate on how to achieve more comprchensive and balanced sampling, guided by covering design principles, to facilitate a more accurate global ranking.
Motivation. Identifying optimal candidates from the available pool is a critical step in the selec- tion of in-context examples. Partial2Global [6] builds a pool of randomly shuffled variants of the alternative set, partitions each variant into multiple non-overlapping sub-sequences and employs a meta-ranker to perform local ranking predictions. Based these local predictions, partial pairwise preferences are inferred and subsequently aggregated into a pairwise indication set, from which a global ranking is derived by solving a least squares problem. While this scheme yields reasonable results, it suffers from two primary limitations:
(1) Incomplete Coverage: Despite generating numerous randomly shuffled variants of Y/ into the obscrvation pool, the random sampling process does not guarantee that all possible pairwise relationships between candidates are captured. Given typically alternative set size K = 50 and a local ranker length k 5 as in Partial2Global, ensuring that every candidate pair is compared within at least one k-length sub-sequence corresponds to constructing a C(50, 5, 2) covering design. In fact, as indicated in Theorem ! a C(50, 5, 2) covering design requires a minimum of 130 distinct k-length sub-sequences to guarantee full pairwise coverage. While random sampling in Partial2Global (i.e., with 50 sub-sequences) might generate numerous sequences, lacks this systematic guarantee and thus may fail to capture all essential pairwise relationships.
(2) Non-Uniform Preference Weighting: Although Partial2Global claims to account for contra- dicting predictions, thereby offering the potential to correct erroneous ones, it does not control for the frequency of repeated pairwise preferences. This can lead to certain locally derived pairwise preferences being overrepresented. thereby adversely influencing the global ranking.
Theorem (Schonheim Lower Bound (27) Considering (K, I) covering design C(K, k, t).
the Schonheim lower bound for such a covering design's 's size is
K K 1
K - 1 1 11.
C(K,
(8)
k-t+1
Covering designs offer a principled approach to mitigate both of these limitations effectively. The inherent combinatorial structure of covering design guarantees exhaustive coverage of all pairwise relationships with an optimally minimal near-minimal number of sub-sequences, while the asso- ciated lower bound promotes balanced and uniform sampling.
When it comes implementation, the pre-computed optimal covering designs allows for an efficient strategy: one can simply generate a randomly shuffled variant of the alternative set V, of size K" and sample k-length sequences from it according to the structure of predefined optimal covering design C* (K', k, t). This structured sampling strategy incurs negligible additional computational overhead while providing a strong guarantee of comprehensive and balanced pairwise coverage.
Experiments
We give the setups, and evaluate our RH-Paarial2Global several visual tasks.
Dataset. Following VPR 2 and Partial2Global (6. we adopt three visual tasks: foreground seg- mentation, single object detection, and image colorization. For the segmentation task, We utilize the Pascal-5 [26] dataset, which comprises four different image splits. Performance is reported using the mean Intersection over Union (mloU) for each split, along with the average mIoU across all four splits. The Pascal VOC 2012 dataset (28) is employed for the single object detection task. Consistent with MAE-VQGAN M. our evaluation subset includes only images containing a single object with Pascal annotations, excluding trivial cases where an object occupies more than 50% of the image area. For the colorization task, we sample a test set from the validation set of ILSVRC2012 [29 to evaluate model performance, using Mean Squared Error (MSE) as the evaluation metric.
Implementation details. Given that our proposed OH--Pariial2llobal method does not require ad- ditional model, we fully adopt the settings of Partial2Global [6]. Specifically, we train meta-rankers with both lengths of 5 and 10 for foreground segmentation and single object detection, while meta-
rankers of length 3 and 5 are utilized for the colorization task. Following VPR, all visual similarity scores in our experiments are computed using the vision encoder of CLIP (30), which was pre-
trained using multimodal contrastive learning. Additionally, we employ DINOv2 [1] as the feature

---

Table 2: Comparison of our proposed RH-Partial2Global with some state-of-the-art VICL methods.
Seg.
(mloU)
Method
Ref.
Det. (mIoU)
Color: (MSE)
Fold-O
Fold-1
Fold-2
Fold-3
Avg.
MAE- VQGAN 1
NIPS' 22
28.66
30.21
27.81
23.55
27.56
25,45
0.67
UnsupPR 2
NIPS' 23
34.75
35,92
32.41
31.16
33.56
26,84
0,63
SupPR 2
NIPS' 23
37.08
38.43
34.40
32.32
35.56
28.22
0,63
Partial2Global 0
NIPS' 24
38.81
41.54
37.25
36.01
38.40
30.66
0.58
RH-Partial ParraiaatGGobball (Ours)
39.25
42.15
38.06
36.60
39.02
30.94
0,56
prompt-SelF
arXiv' 2023
42.48
43.34
39.76
38.50
Purtaal2Globll
NIPS'22 24
43.23
45.50
41.79
40.22
42.69
43.53
45.88
41.99
40.90
43.08
33.28
Partial2Glohal
0.00
5.20
16.78
41.47
17.41
Ours
36.31
44.66
66.43
72.01
67.49
Figure 1: Qualitative comparison between our proposed RH-Partia/2Global and Partial2Global in the foreground segmentation task. For each comparison item, display the image grid following the input order of MAE-VQGAN the first row contains the in-context example alongside its corre- sponding label, while the second shows the query image and its predicted result. The IoU value is reported below each image grid to facilitate performance evaluation.
extractor and optimize using the AdamW optimizer with a learning rate of x 10-5 and batch size of 64. For our conformal prediction-based selection strategy, while acknowledging its sensitivity to the chosen quantile, we consistently set a 0.85, corresponding to an 85% confidence level across all tasks, and adopt the negative KL Divergence our proposed 2H-Partial2Global against five previous methods: MAE VQGAN, the unsupervised and supervised variants of VPR. the original Partial2Global framework, and prompt-SelF [II), method primarily characterized by its ensemble- based strategy. For fair comparison, results for RH-PaattillGlooal presented both with and without the test-time voting ensemble.
conformity function. We benchmark
4.1
Main results
The quantitative results for the foreground segmentation, object detection and image colorization tasks presented in Table Here several observations. First of all, RH-Partial2Global demon- strates consistent performance improvements across all visual tasks compared to the baseline, which supports the efficacy of our proposed reliable selection and holistic sampling strategies. For exam- ple, on the third fold of Pascal5', RH-Partial2Glloal outperforms pure Partial2Global by 0.81 % and achieves an average increase of 0.62% across all four folds. Although these increments not uni- formly large, they are particularly noteworthy due to their consistent improvements across all cases without additional model training, and their confirmed statistical significance. Secondly, a nuanced observation is that the performance improvements on Fold-O and Fold-3 are not significant as those observed on Fold-1 and Fold-2. This disparity might be attributable to the characteristics of conformal prediction that its effectiveness in reliably predicting interval or set typically benefits from a sufficiently large calibration set. As indicated in Table [] Fold-0 and Fold-3 contain considerably fewer samples than Fold-1 and Fold-2, potentially impacting the robustness of the reliability assess- ment. Lastly, the pattern of improvement remains consistent whether a test-time voting strategy employed or not, further underscoring the generalization capability of our proposed method.

---

Table 3: Average top-k oracle in-context learning performances of the initial and refined alternative set on the segmentation task, which represented by "X" and " "V", respectively. denotes the difference between the two performances.
Fold-0
Fold-1
Fold-2
Fold-3
Avg.
Top-k
X
45.93
45.68
-0.25
49.78
49.61
-0.1
46.54
46.18
-0.26
44.63
44.38
-025
46.72
46,46
-0.26
10
43.82
43.58
-0.27
47.42
47.27
-0.15
44.09
43.73
-0.36
41.49
41.27
-0.22
44.21
43.96
-0.25
15
42.24
42.01
-0.23
45.70
45.56
-0.14
42.22
41.83
-0.39
39.27
39.03
-0.24
42.36
42.11
-0.25
Table The impact of different strategies on the segmentation task. Sep. Sed, and represent proposed conformal prediction-guiet candidate selection strategy, covering design-based sampling strategy and auxiliary filling strategy, respectively.
repressennnt our
Strategy Sed
Seg. (mIoU) Fold-2
Sep
Sym
Fold-O
Fold-1
Fold-3
Avg.
(a)
X
X
x
38.81
41.54
37.25
36.01
38.40
(h)
/
X
39.05
41.89
37.81
36.35
38.78
(c)
39.15
41.93
37.75
36.32
38.79
(d)
39.25
42.15
38.06
36.60
39.02
(0)
39.36
42.54
38.45
36.72
39.27
In order to further illustrate the superiority of RH- ParraiallllGllb,,, Figure 1 presents comparative visualizations of in-context examples selected by our method versus Partial2Global for the segmen- tation task, alongside their resulting segmentation outputs. A key observation is that, compared to Partial2Global, the prompts chosen by EEH-Partial2Globbl generally exhibit not only high categori- cal relevance to the query but also greater alignment in terms of object pose, scene context, and other fine-grained visual attributes. For instance, when presented with a query image a dog (fourth ex- ample in Fig. []. both methods select dog an in-context prompt. However, RH-Partial2Global selects an image where the dog's pose precisely mirrors that of the query. Similarly, in the last vi- sualized instance, CH-Partial2Global selects a prompt featuring long dining table with similar orientation to the query, while Partial2Global opts for a square table, a choice that is semantically re- lated but structurally less analogous. These examples suggest that RH- Partial2Global is more adept at identifying in-context examples with high spatial and structural similarity to the query, which enhances their reliability and likely contributes to its superior performance across various tasks.
4.2 Ablation study
In order to fully validate the effectiveness of RH-PaaliaalGloobl, we conduct a series of ablation studies on it. All experiments this section are performed the foreground segmentation task.
Whether RH-Partial2Global really preserves reliable examples with good quality? A key ob- jective is to validate whether our proposed conformal prediction-based selection strategy effectively preserves reliable, high-quality in-context examples while discarding less suitable ones. While our strategy is designed to ensure that selected examples meet pre-defined reliability threshold, we empirically verify its impact on the qualities of the selected examples for the test set. The refined sets are obtained using our selection strategy with a confidence level = 0.85, which results in approximately 15% of the initial candidates being discarded as less reliable. As shown in Table we evaluate the average loU achieved by the top-5, top-10, and top- 15 highest-quality examples from both the initial and the refined alternative sets. These results reveals that the performance up- per bound, indicated by the average IoU of these top-k examples, remains remarkably stable, which achieved despite discarding approximately 15% of the initial candidates. This outcome strongly suggests that our conformal prediction-based selection strategy effectively preserves high-quality, reliable examples while successfully filtering out sub-optimal ones.
Visualization of scatter plot with regression line for similarity and IoU scores. To intuitively illustrate the relationship between visual similarity scores and loU scores when specific sample serves as an in-context prompt for others, we employ scatter plots with fitted linear regression lines. preseetss several such illustrative examples. Across these visualizations, the p-values associ- ated with the linear regression analyses are consistently below the 0.05 significance level. However, the magnitudes of the regression slopes are predominantly modest. These observations indicate that
8

---

Eectudy
Rendurry
Figure 2: Visualization of scatter plot with regression line of similarity and IoU scores.
Table 5: Average top-k oracle in-context learning performances of the initial and filled alternative set on the segmentation task, which is represented by "x" and "+", respectively. "_" denotes the difference between the two performances.
Fold-0
Fold-1
Fold-2
Fold-3
Avg.
Top-k
X
45.93
46.04
+0.11
49.78
50.66
+0.88
46.54
46.82
+0.28
44.63
44.96
+0.33
46.72
47.12
+0.40
43.82
44.05
+0.23
47.42
48.36
+0.94
44.09
44.47
+0.38
41.49
41.96
+0.47
44.21
44.71
+0.50
15
42.24
42.56
+0.32
45.70
46.70
+1.00
42.22
42.70
+0.48
39.27
39.81
+0.54
42.36
42.94
+0.58
while a linear trend between visual similarity and IoU performance often exists, the strength of this linear correlation is generally weak. These visual findings corroborate the statistical conclusions from the hypothesis tests detailed in Section 3.1 further underscoring the limited predictive power of raw visual similarity for determining prompt effectiveness.
Discussions on the effect of each strategy. To verify the individual and combined efficacy of our proposed conformal strategy Sed, we conducted ablation experiments evaluating different configurations of these com- ponents. As shown in Table 4 (c.g., lines and d), applying either Sep or Sea individually yields performance improvements over the baseline Partial2Global. Moreover, it can be seen in Table that the simultaneous use of these two strategies fails to achieve a synergistic effect. A potential explanation for this observation is that S.p can occasionally reduce the number of reliable candi- dates to fewer than the ranker length k. Under such circumstances, Seul loses its intended utility, as any sampling can ensure complete coverage of all pairwise preferences. Therefore, we introduce an auxiliary filling strategy Syn, selecting the most similar candidates from the constructed reliable set Vo to the query sample Ki for filling. As presented in the line (e) of Table this strategy can further enhance the performance of our PallPaatill2Glohal. We attribute this enhancement to an improved performance upper bound. Similarly, we directly test all prompts from the original and the filled alternative set for each query in the segmentation task and present the average performance of the top-5/10/15 best examples in Table [5] The improvements in this oracle performance, as shown in Table 5
inliicttonnguideed selection strategy Sep and covering design-based sampling
lend strong support
viewpoint.
Universality of conformal prediction-guidee selection strategy. The souillaitt-prioriii assump- tion, which our proposed conformal prediction-guie guided selection strategy Sep aims to address, rep- resents a common limitation in many contemporary Visual In-Context Learning (VICL) methods, extending beyond just the Partial2Global framework. To demonstrate the universality and general- ization capacity of Sep. we therefore apply it to both the unsupervised (UnsupPR) and supervised (SupPR) variants the VPR framework 2] Recognizing that S.p. applied alone might occasionally reduce the number of reliable candidates to even zero, we implemented and evaluated two aug- mented approaches to ensure a viable candidate pool when applying our strategy to VPR: (1) Sep with initial supplementation (referred to as "+Sop" "). If the refined alternative set is empty, it is sup- plemented by selecting the most visually similar candidate from the initial alternative set provided by VPR; (2) S.p combined with the auxiliary filling strategy Syu (referred to "+Su"+ If the resulting alternative set is empty, it is augmented by selecting the most visually similar candi- date from the reliable global set Va- We utilize similarity scores obtained from VPR's pretrained and fine-tuned metric network respectively in (2). The confidence level a is set as 0.55 across all the four folds in the segmentation task. We present all the comparison results in Table
Here are several observations. Firstly, our proposed conformal cuddiciionguudde selection strategy can also demonstrate significant and consistent performance gains across all the folds when com- pared to the baseline VPR. This outcome strongly validates the effectiveness and broader universal- ity of Sep- Moreover, our auxiliary filling strategy Stu is shown to further enhance the performance of VICI. underscoring the quality of our reliable prompt set identified by Sep.

---

Table 6: Performance comparison of VPR variants (UnsupPR and SupPR) with and without the integration of Sep and its augmentation Stslt on the segmentation task.
Seg. (mloU) f
Method
Ref.
Fold-O
Fold-1
Fold-2
Fold-3
Avg.
UnsupPR 2
2IPS'23
34.75
35.92
32.41
31.16
33.56
UnsupPR+SSr
36.07
38.50
35.03
33.53
35.78
UnsupPR+Scp+55pp
36.97
38.80
34.62
33.66
36.01
SupPR 2
NIPS'23
37.08
38.43
34.40
32,32
35.56
SupPR+Scp
37.69
39.45
36.61
33.96
36.93
SupPR+S.p
37.83
39.85
36.61
33.96
37.06
UnsupPR Sp Sa with Different Selection of
SupPR
+ Sk Sye with Different Selection of
6.11
375
(a) Sp
(b) Spp
Performance trends of VPR when enhanced by the Sep and Systt strategies, evaluated across range of parameter values. The setting 'a 1.00' represents the baseline performance of the original VPR methods without these proposed strategies.
Figure 3:
Figure further details the performance of SupPR and UnsupPR when integrated with our Sep and Sym strategies, evaluated under various settings of the conformal prediction parameter It is evident from these results that regardless of the specific a value selected, applying our strategies enables both VPR variants consistently outperform the original VPR framework. Such consistent outperformance, even with varying a configurations, strongly demonstrates the universality and robustness of our conformal prediction-guides selection strategy.
Conclusion
This paper introduces RH--aatial2Global, an enhanced variant Partial2Global, tailored for in- context example selection in Visual In-Context Learning (VICL). Specifically, we first challenge the default similarity-priorii assumption that an image more similar to the query image is more suitable in-context example, and further validate its inherent limitations from statistical perspective. Subsequently, we develop a sample selection strategy based on jackknife conformal prediction to refine the alternative set, which is initially established following the cimillartt-prririin assumption, thereby retaining only reliable candidate samples. Furthermore, we propose covering design- based sampling strategy to supersede the random operations Partial2Global, facilitating a more comprehensive and balanced construction of pairwise preference relationships. By integrating these two strategies, our RH-Pat Partial2Global yields improved global ranking predictions, paving the way for more reliable and holistic VICL prompt selection. Extensive experiments across multiple visual tasks demonstrate that Par-aariialGGloball not only outperforms its predecessor, Partial2Global, but also consistently achieves excellent performance.
Limitations. While our proposed method significantly enhances Partial2Global, its reliable set pre- diction is sensitive dataset size, potentially limiting gains with scarce data due reduced statis- tical robustness. Nevertheless, we contend that our reflections on the similarity-prione and exploration based on conformal prediction hold broader significance for VICL. This insight can encourage the development of example selection strategies that move beyond mere similarity, potentially advancing performance across diverse models and tasks.
assumption
10

---

Acknowledgments
This research is partly supported by the Hubei Provincial Natural Science Foundation of China under Grant No. No.2022CFA055.
References
[1] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. In Advances in Neural Information Processing Systems, vol- ume 35, pages 25005-25017, 2022.
[2] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in- context learning? In Advances in Neural Information Processing Systems, volume 36, pages 17773-17794, 2023.
[3] Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang, Mingyuan Zhou, et al. In-context learning unlocked for diffusion models. Advances in Neural Information Processing Systems, 36:8542-8562, 2023.
[4] Yucheng Zhou, Xiang Li, Qianning Wang. and Jianbing Shen. Visual in-context learning for large vision-language models. arXiv preprint 15544 2024.
[5] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 6830-6839, June 2023.
[6] Chengming Xu, Chen Liu, Yikai Wang, Yuan Yao, and Yanwei Fu. Towards global optimal visual in-context learning prompt selection. In Advances in Neural Information Processing Systems, volume 37, pages 74945-74965. Curran Associates, Inc., 2024.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language mod- els are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[8] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Ad- vances in neural information processing systems, 36, 2024.
[9] Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-Im: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305. /2223.....22..... 2023.
[10] Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.12740, 2023.
[11] Yanpeng Sun, Qiang Chen, Jian Wang, Jingdong Wang, and Zechao Li. Exploring effective factors for improving visual in-context learning. arXiv preprint arXiw:2304.04748, 2023.
[12] Glenn Shafer and Vladimir Vovk. A tutorial conformal prediction. Journal of Machine Learning Research, 9(3), 2008
[13] Xiaofan Zhou. Baiting Chen, Yu Gui, and Lu Cheng. Conformal prediction: A data perspec- tive. arXiv preprint arXiv:2410.06494, 2024.
[14] Sophia Sun. Conformal methods for quantifying uncertainty in spatiotemporal data: A survey. arXiv preprint wtX.::22099035800 2022.
[15] Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan Tibshi- rani. Class-conditional conformal prediction with many classes. Advances in neural informa- tion processing systems, 36:64555-64577 2023.

---

[16] Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking Ilms via uncertainty quantification. Advances in Neural Information Processing Systems, 111555--53388 2024.
[17] Jiankai Sun, Yiqi Jiang. Jianing Qiu, Parth Nobel, Mykel J Kochenderfer, and Mac Schwa- ger. Conformal prediction for uncertainty-awant planning with diffusion dynamics model. Advances in Neural Information Processing Systems, 36:80324 8033-80337,1 2023.
[18] Roberto I Oliveira, Paulo Orenstein, Thiago Ramos, and João Vitor Romano. Split conformal prediction and non-exchangeable data. Journal of Machine Learning Research, 25(225):1-38, 2024.
[19] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-fiee predictive inference for regression. Journal of the American Statistical Asso- ciation, 113(523):1044 1010--1191,111111111111111111111 2018.
[20] Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, 49())486-5077 2021.
[21] Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74:9-28, 2015.
[22] Kfir M Cohen, Sangwoo Park, Osvaldo Simeone, and Shlomo Shamai Shitz. Cross-validation conformal risk control. In 2024 IEEE International Symposium Information Theory (ISIT), pages 250-255. IEEE, 2024.
[23] Anastasios N Angelopoulos, Stephen Bates, Emmanuel J Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:21 10.01052, 2021.
[24] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6):1-34, 2021.
[25] Daniel M Gordon, Oren Patashnik, and Greg Kuperberg. New constructions for covering designs. Journal of Combinatorial Designs, 3(4):269 284, 1995.
[26] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv: 1e1.1179911111 2017.
[27] Johanan Schonheim. On coverings. Pacific Journal of Mathematics, 14(4):1405-1411 1964.
[28] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 11 111:98-136. 2015.
[29] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bemsstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11::11-252,, 2015.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[31] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction claims that introduces an enhanced variant of Partial2Global for reliable and holistic selection of in-context examples for VICL, which is consistent with our main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: we have discuss the limitations of our work Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification:
Our proposed method does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided all the experimental setups and implementation details for reproducibility in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Our results do not rely any private data. The instructions provided in the paper are sufficient to reproduce the experimental results. We will provide open access to the code upon acceptance of this paper.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips. cc/
ptblic//guides/DddSSuuiiiis ssiocillcyyyyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips.c//ppp htt.cc/ppulicigggdeeeeeee htte://ueeeeeeeeeeiiieeeeee issii onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification: We have provided all the experimental setups and implementation details in Sec. 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: We cannot provide error bar for some competitors. Besides, the current results are sufficient to show the efficacy of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provide sufficient information on the computer resources in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpt:t//turttippee http:///pcpppppttcecccccceeeeeee
Answer: [Yes]
Justification: Our research focuses on general visual tasks, which totally conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal im- pacts of the work in the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite all used public datasets and pre-trained models in Sec.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, wwtppppppppphtoooo
com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: We do not include any crowdsourcing experiments and research with human subjects our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We only conduct objective experiments in our paper.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This work does not incorporate Large Language Models (LLMs) into its core methodology. Consequently, no description of such usage is provided.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy htttpp://oorrps.cc//oooeerreeeee for what should or should not be described.
19

---

A Addition ablation study and discussion
Analysis of confidence level for RH-Partial2Global To further demonstrate the robustness of our proposed strategy with respect the confidence level a, we conduct a corresponding ablation study on RH-Partial2Global evaluating its performance with values ranging from 0.94 to 0.80. The results are presented Figure The results demonstrate that all tested a values yield a perfor- mance improvement over the baseline (i.e. 1.00).
RHI-PatialZCilobal Differett Selection
Folds
Toubs
12
500 Scare
Figure 4: Performance trends of PH-Partial2Glohal evaluated across range of a parameter values. The setting 1.00' represents the baseline performance of the pure Partial2Glohal method.
The selection of our conformity function. for our strategy. we evaluate both Spearman correlation and negative KI. Divergence beyond upon the Partial2Global framework, comparing their respective impacts on overall performance. The comparison results are shown in Table 7
identify the more effective conformity function
Table Comparison of our proposed CH-Paatial2Glooal with Spearman correlation or negative KL.
Divergence as the conformity function.
Seg. (mIoU) Fold-2
Method
Ref.
Fold-O
Fold-1
Fold-3
Avg.
Partial2Glohal
NIPS'24
38.81
41.54
37.25
36.01
38.40
RlloaalialGGlohall (Cor) RH-Partial RH-PPartia2Glohal 2Glohal (KL.)
39.13
41.88
37.57
36.48
38.77
39.25
42.15
38.06
36.60
39.02
Experimental results on the Pascal-5' dataset indicate that negative KL Divergence consistently yields superior segmentation performance compared Spearman correlation across all the four folds. Based on this finding, negative KL Divergence is adopted as the definitive conformity function for our main experiments. This observed advantage can be attributed the fundamental differences in how these metrics capture data characteristics. Spearman correlation, which quantifies rank- based monotonic associations, primarily reflects ordinal structure. Consequently, it less sensitive to the detailed distributional information embedded in specific value magnitudes and higher-order statistical moments of the underlying data. In contrast, KI. Divergence assesses the dissimilarity between entire probability distributions, thereby inherently accounting for discrepancies across all orders of moments and offering a more comprehensive comparison of distributional differences.
Visualization of single object detection task. Figure 3 presents comparative visualizations of in- context examples selected by our method versus Paltial for the detection task, alongside their resulting detection outputs. Consistent with findings from the segmentation task, prompts selected by CH-Paariia2Gloobl generally exhibit superior alignment with the query in terms of scene context and other fine-grained visual attributes when compared to those from Partial2Global. For instance, as illustrated in the central column of comparative visualizations, the scene context within the prompts chosen by our RH-PaariilZGlooal demonstrates a notably higher degree of similarity to the query sample's context.
20

---

Partial2Globa
0.00
0.00
0.00
0.00
6.99
38.14
28.63
7.13
22.43
28.30
Ours
49.00
55.49
33.87
29.19
46.82
79.44
58.03
35.64
74.79
51.41
Figure 5: Qualitative comparison of single object detection performance between our RH- Partial2Global and Partial2Global. To enhance visual clarity and simplicity, bounding boxes overlaid directly onto the images, rather than displaying the complete image grids. Each depicted item consists of an in-context example (left) and the corresponding query image (right).
Why the selection strategy is jackknife conformal prediction-guideti The underlying reasons can be summarized as follows. Firstly, mirroring the lcave-onc-out strategy of jackknife, our se- lection strategy individually evaluates each candidate by assessing its performance when used as a prompt for the other samples. Secondly, similar to jackknife conformal prediction, our selection strategy emphasized maintaining reliability across the alternative gle sample. Lastly, instead of pinpointing a single best prompt, it selects a set of reliable prompts, aligning with conformal prediction's 's principle of "set prediction". Therefore, our selection strategy can be regarded as a jackknife conformal prediction-guided method.
rather than optimizing for sin-
B Broader impact
The proposed work itself is not anticipated lead to significant direct negative social impacts. Despite of this, it explicitly acknowledges and discusses a critical broader concern: data bias. The paper notes that if the data used for in-context learning (or any data-driven AI model) contains existing societal biases, the models leveraging this data (including those incorporating the proposed enhancements) could inadvertently perpetuate even amplify these biases. This could potentially lead to unfair or discriminatory outcomes, especially in sensitive application areas.
21


Reviews:
[{"review_id": "SRXAMEEvh2", "reviewer": "Reviewer_B3W8", "summary": "This paper introduces RH-Partial2Global, an enhanced version of the Partial2Global framework for visual in-context learning (VICL) prompt selection. The authors identify and aim to address two key limitations in prior work: the unsubstantiated \"similarity-priority assumption\" and the potentially incomplete and redundant nature of random sampling for generating pairwise preferences. To tackle these issues, the paper proposes a two-pronged approach. First, it introduces a jackknife conformal prediction-guided strategy to create a more reliable set of candidate examples, statistically questioning the simple heuristic that visual similarity equals better performance. Second, it replaces random sampling with a covering design-based strategy to ensure a more holistic and uniform coverage of pairwise comparisons. The authors conduct experiments on foreground segmentation, object detection, and colorization tasks, demonstrating that RH-Partial2Global consistently outperforms the original Partial2Global, with and without a test-time voting ensemble.", "questions": "See weakness", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Pros\n\n1. The paper's primary strength lies in its critical re-evaluation of a foundational, yet under-examined, assumption in the VICL field. The \"similarity-priority assumption\" is intuitive but, as the authors demonstrate, not always robust. By conducting a formal statistical analysis using Spearman's rank correlation, the paper provides quantitative evidence that while a monotonic relationship between similarity and task performance exists, its strength is often weak. \n\n2. The proposed solutions are well-motivated and technically sound. The use of jackknife conformal prediction to construct a reliable set of candidates is a principled approach to mitigate the identified weakness in the similarity-priority heuristic. It provides a statistical, data-driven method for filtering the initial, similarity-based candidate pool, aiming to improve the quality of inputs for the ranking process.\n\n3. The experimental results, though showing modest gains in some cases, are consistently positive across multiple tasks and evaluation metrics. The fact that RH-Partial2Global achieves better performance without requiring additional model training is a significant advantage, highlighting the efficiency of the proposed enhancements. The ablation studies are also well-designed, effectively isolating the contributions of the conformal prediction and covering design strategies and providing a clear rationale for the inclusion of the auxiliary filling strategy.\n\nCons:\n\n1. While the paper's challenge to the similarity-priority assumption is a key strength, the subsequent methodological choice to still use similarity as the primary basis for the initial alternative set (Y_q ) seems somewhat contradictory. The algorithm first constructs an alternative set based purely on top-K similarity and then filters it. This raises the question of whether a more effective initial set could be constructed using a different, or hybrid, criterion from the outset. The paper could be strengthened by discussing why this \"filter-down\" approach is preferred over a more integrated method for generating the initial candidate pool.\n\n2. A second weakness lies in the discussion of the performance gains. The improvements, particularly in some folds of the segmentation task, are marginal (e.g., 0.44% on Fold-0 for RH-Partial2Global without voting). While the authors correctly argue that consistency and statistical significance are important, a deeper analysis of why the gains are not more substantial would be beneficial. Is the performance of the baseline Partial2Global already near the ceiling for these tasks with the given backbone models, or are there other, more significant bottlenecks that the proposed methods do not address?\n\n3. Furthermore, the paper notes that the effectiveness of conformal prediction can be sensitive to the size of the calibration set , as suggested by the lower performance gains on the smaller folds (Fold-0 and Fold-3). This is a critical limitation that is mentioned but could be explored more deeply. The authors set the confidence level α to 0.85 across all experiments. An analysis of how performance varies with different values of α, and how this interacts with dataset size, would provide valuable insights into the robustness and practical applicability of the proposed method.", "was_revised": true, "final_justification": "After reading the rebuttal, all my questions have been properly addressed. Therefore, I raise my score.", "rebuttal": "### **For Con 1: Regarding the \"filter-down\" mode**\n\nWe sincerely thank the reviewer for this insightful and important question. We would like to clarify the multi-faceted rationale behind our design choice. \n\n1. **Our central argument is not that similarity is useless, but that similarity alone is an insufficient criterion.** As our analysis in Table 1 demonstrates, a general association between similarity and quality often exists statistically, even if it is not always strong. This makes a high-similarity set a statistically sound and efficient starting point to work from. It allows us to efficiently narrow the vast universe of possibilities down to a smaller, more relevant candidate pool, without losing promising options.\n2. **This \"filter-down\" mode ensures high generalizability.** A key goal of our work is to propose a method that is applicable across diverse tasks. Designing unique, \"integrated\" or \"hybrid\" metrics for each specific task would be exceptionally complex and would result in a less universal, more brittle solution. Our current design provides a task-agnostic and elegant principle that maintains its power across different applications.\n3. In fact, the \"filter-down\" method itself functions as a powerful and interpretable hybrid criteria. For one thing, it enforces generalized quality by ensuring that selected prompts are drawn from a reliable set with proven overall performance, independent of any individual test query. For another, it establishes query-specific relevance by using similarity to form the initial candidate pool. **The intersection of these two sets allows us to obtain prompts that are not only contextually appropriate for the query but also robustly effective.**\n\n------\n\n### **For Con 2: Regarding the performance gains**\n\nWe thank the reviewer for this excellent and critical question. Our analysis suggests that the magnitude of the performance gain is influenced by several factors.\n\n1. **We would like to highlight that our method's impact varies depending on the baseline it is applied to.** As detailed in the **Table 1 of Supplementary Material**, our strategy yields substantial improvements when integrated with the VPR variants, where we observed gains ranging from **1.5% to 3%**. This demonstrates the general efficacy and high potential of our proposed method under different conditions.\n\n2. The modest gains on Partial2Global can stem from three primary factors:\n\n   **(i) In our initial main experiments, we deliberately used a fixed confidence level ($\\alpha=0.85$) across all tasks.** This was to demonstrate the \"out-of-the-box\" utility and ease of use of our method without task-specific tuning. To test the hypothesis that gains could be larger with a tuned $\\alpha$, we conduct a new experiment as follows. \n\n   |    Fold-0     |    Fold-1     |    Fold-2     |    Fold-3     | Avg. |\n   | :-----------: | :-----------: | :-----------: | :-----------: | :-----: |\n   | $\\alpha=0.85$ | $\\alpha=0.85$ | $\\alpha=0.85$ | $\\alpha=0.85$ |   --    |\n   |     39.25     |     42.15     |     38.06     |     36.60     |  39.02  |\n   | $\\alpha=0.82$ | $\\alpha=0.92$ | $\\alpha=0.90$ | $\\alpha=0.86$ |   --    |\n   |     39.48     |     42.39     |     38.28     |     36.81     |  39.24  |\n\n   The results show that when $\\alpha$ is selected on a per-dataset basis, the performance gains on Partial2Global become more significant. \n\n   **(ii) There is a subtle interaction between our CP-based selection strategy and the internal aggregation process of Partial2Global, which translates local rankings into a global ranking.** Our selection method can occasionally reduce the diversity of local ranking information available to the baseline, slightly impeding its aggregation mechanism. As described in Table 4 and ablation study, we introduced a targeted \"filling strategy\" as a compensatory measure, which enriches the candidate set by adding the most similar reliable prompts. The performance increase observed after applying this strategy strongly supports our hypothesis.\n\n   **(iii) It is crucial to consider the limitations of the foundational model.** MAE-VQGAN, while a standard backbone for VICL research methods like VPR and Partial2Global, is an earlier-generation architecture. Therefore, although RH-Partial2Global is highly optimized, it operates near the ceiling for MAE-VQGAN.\n\n   **To validate our opinion, we standardize the backbone architecture by replacing MAE−VQGAN with SegGPT [1], a more advanced model in both Partial2Global and our proposed RH-Partial2Global.** Since the prompt selection mechanism in SegGPT  involves randomly sampling images that share the same category label from the training set of PASCAL-5i as the prompt image (as illustrated in its Appendix A), we contend that this approach constitutes a form of information leakage, as it implicitly provides the model with class-level knowledge. Consequently, to create a stricter and more fair baseline, we utilize the most similar image as the prompt and evaluate under our settings. This allows the evaluation to focus solely on the merits of our proposed strategies. The quantitative results for segmentation task are presented as follows.\n\n   |           Method           | Fold-0 | Fold-1 | Fold-2 | Fold-3 | Avg.  |\n   | :------------------------: | :----: | :----: | :----: | :----: | :---: |\n   |           SegGPT           | 64.35  | 66.79  | 58.91  | 65.68  | 63.93 |\n   |  Partial2Global (SegGPT)   | 66.52  | 71.55  | 62.48  | 68.74  | 67.32 |\n   | RH-Partial2Global (SegGPT) | 69.45  | 74.35  | 66.33  | 71.90  | 70.51 |\n\n   Notably, by integrating a more advanced generative model, the advantages of our method become substantially more pronounced.\n\n   [1] SegGPT: Segmenting Everything In Context\n\n------\n\n### **For Con 3: Regarding the analysis of confidence level**\n\nWe sincerely thank the reviewer for this critical and highly constructive point. We agree that a deeper exploration of our method's sensitivity to the set size, and its interaction with the confidence level $\\alpha$, is essential for understanding its practical applicability.\n\n1. **To demonstrate the robustness of our proposed selection strategy with respect to the confidence level $\\alpha$, we present the following ablation studies on VPR and Partial2Global.** We would like to draw the reviewer's attention to the **Supplementary Material (Figure 1)**. This study, conducted on the VPR baseline, shows that our strategy consistently delivers performance improvements across a wide range of $\\alpha$ values (from 0.5 to 0.95). These results as well as those obtained **on RH-Partial2Global below** establish the general effectiveness of our core approach. *(Note: We used VPR for this specific analysis because its architecture provides a cleaner and more direct assessment of $\\alpha$'s impact. This avoids the potentially confounding effects from Partial2Global's ranking aggregation mechanism, allowing for a more isolated analysis of the hyperparameter itself.)*\n\n   | $\\alpha$ | 1.00  | 0.94  | 0.93  |   0.92    | 0.91  |   0.90    | 0.89  | 0.88  | 0.87  |   0.86    | 0.85  | 0.84  | 0.83  |   0.82    | 0.81  | 0.80  |\n   | :------: | :---: | :---: | :---: | :-------: | :---: | :-------: | :---: | :---: | :---: | :-------: | :---: | :---: | :---: | :-------: | :---: | :---: |\n   |  Fold-0  | 38.81 | 39.19 | 39.35 |   39.47   | 39.02 |   39.11   | 39.44 | 39.10 | 39.03 |   39.24   | 39.25 | 39.08 | 39.12 | **39.48** | 39.32 | 39.36 |\n   |  Fold-1  | 41.54 | 42.28 | 42.17 | **42.39** | 42.09 |   42.31   | 42.36 | 42.14 | 42.14 |   42.19   | 42.15 | 42.20 | 42.31 |   42.09   | 42.24 | 42.02 |\n   |  Fold-2  | 37.25 | 38.25 | 38.11 |   38.25   | 38.12 | **38.28** | 37.86 | 37.99 | 38.08 |   38.24   | 38.06 | 37.99 | 38.05 |   37.93   | 38.01 | 38.09 |\n   |  Fold-3  | 36.01 | 36.58 | 36.52 |   36.57   | 36.27 |   36.51   | 36.30 | 36.70 | 36.48 | **36.81** | 36.60 | 36.57 | 36.76 |   36.46   | 36.47 | 36.35 |\n\n2. **Regarding the interaction between $\\alpha$ and dataset size, we provide a principled analysis grounded in the theory of conformal prediction: The effectiveness of conformal prediction relies on the exchangeability of data, which is more robust with larger datasets.** With smaller sets (as in our Fold-0 and Fold-3), two issues may arise:\n\n   (i) The empirical distribution of conformity scores becomes more coarse-grained, making the quantile estimation less precise. \n\n   (ii) The set of scores is more susceptible to the influence of outliers or sampling noise, which can destabilize the calibration process. \n\n   Based on this theoretical understanding, we can offer a clear recommendation. For applications involving smaller calibration sets, we advise selecting a lower confidence level $\\alpha$. A lower α leads to a smaller, more selective reliable set. This acts as a safeguard against the potential instability, ensuring that only the most reliable candidates are included. The optimal $\\alpha$ for each fold, as presented in the table above, further substantiate our recommendation."}, {"review_id": "yHiosNSXXS", "reviewer": "Reviewer_LDet", "summary": "This paper introduces an enhanced variant of Partial2Global designed for the reliable and holistic selection of in-context examples in Visual In-Context Learning. Specifically, this paper leverages a jackknife conformal prediction-guided strategy to construct reliable 16 alternative sets and a covering design-based sampling approach to ensure comprehensive and uniform coverage of pairwise preferences.", "questions": "I have one question (no need to address it in the response). While Visual In-Context Learning is an intriguing task, many general-purpose models (e.g., SegGPT) already demonstrate strong capabilities in this domain due to their training on large-scale data. Given this context, what is the necessity of studying Visual In-Context Learning on small datasets like Pascal-5i or Pascal VOC 2012?\n\nAdditionally, for larger models, such as those described in \"Sequential Modeling Enables Scalable Learning for Large Vision Models\", does investigating prompt selection strategies become more critical compared to small-data experiments? The authors should clarify whether their work can be used for existing large-scale models.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is well-motivated and well-written.\n\nAn enhanced variant of Partial2Global is designed for the reliable and holistic selection of in-context examples in VICLl.\n\nThe experimental results across diverse visual tasks show excellent performance.\n\nWeaknesses:\n1. More general tasks: SegGPT[1] is a generalist model for segmenting everything in Visual In-Context\nLearning. Can RH-Partial2Global be used in this task?\n\n2. In Table 2, can Painter and SegGPT be used in this dataset?\n\n[1] SegGPT: Segmenting Everything In Context\n[2] Sequential Modeling Enables Scalable Learning for Large Vision Models", "was_revised": true, "final_justification": "An enhanced variant of Partial2Global is designed for the reliable and holistic selection of in-context examples in VICLl. The experimental results across diverse visual tasks show excellent performance. My concerns are well solved in the rebuttal. After considering other reviews and the corresponding answers, I'd like to keep the rating at the current stage.", "rebuttal": "### **For Question 1: Regarding VICL on small datasets**\n\nWe sincerely thank the reviewer for these insightful, high-level questions. We completely agree that large-scale, general-purpose models like SegGPT have demonstrated truly impressive capabilities in VICL. We believe it is a valuable opportunity to clarify the specific research gap our work aims to fill and the broader context in which it sits.\n\n**Rather than focusing on training a massive, end-to-end model, our work, along with related methods like VPR and Partial2Global, formulates VICL as a prompt selection problem, which is also a critical issue for general-purpose models.** From this perspective, the role of smaller, standardized datasets like Pascal-5i and VOC 2012 becomes critical, but in a different capacity. These datasets are not used to train the foundational models (our base inpainting model, MAE-VQGAN, was pre-trained on a larger dataset). Instead, they serve as controlled, reproducible **benchmarks**. Their purpose is to enable a rigorous and fair evaluation of the prompt selection algorithms themselves, allowing the research community to isolate and compare the efficacy of different selection strategies.\n\n----\n### **For Weaknesses and Question 2: Regarding the applicability of our work to large-scale models like SegGPT**\n\nWe thank the reviewer for this insightful question regarding the applicability of our work to large-scale models. This is a critical point that touches upon the scalability and future relevance of our framework\n\n**In fact, our method treats the generative or in-context learning model as a black box, where the generative model in our framework, MAE-VQGAN is a modular component for fair comparison, which could be readily swapped for any state-of-the-art inpainting model.** The fundamental question we address, \"how does one best select visual prompts to elicit a desired behavior from a pre-trained model through its in-context learning ability?\", remains crucial and unsolved, regardless of whether the base model is MAE-VQGAN or a future large-scale foundation model. \n\n**To validate our opinion, we standardize the backbone architecture by replacing MAE−VQGAN with SegGPT in both the Partial2Global baseline and our proposed RH-Partial2Global framework.** Since the prompt selection mechanism in SegGPT involves randomly sampling images that share the same category label from the training set of PASCAL-5i as the prompt image (as illustrated in its Appendix A), we contend that this approach constitutes a form of information leakage, as it implicitly provides the model with class-level knowledge. Consequently, to create a stricter and more fair baseline, we utilize the single most similar image as the prompt, selected without prior category knowledge, and evaluate under our settings. This allows the evaluation to focus solely on the merits of our proposed strategies. The quantitative results for segmentation task are presented as follows.\n\n|           Method           | Fold-0 | Fold-1 | Fold-2 | Fold-3 | Avg.  |\n| :------------------------: | :----: | :----: | :----: | :----: | :---: |\n|           SegGPT           | 64.35  | 66.79  | 58.91  | 65.68  | 63.93 |\n|  Partial2Global (SegGPT)   | 66.52  | 71.55  | 62.48  | 68.74  | 67.32 |\n| RH-Partial2Global (SegGPT) | 69.45  | 74.35  | 66.33  | 71.90  | 70.51 |\n\nNotably, when integrated with an alternative and more advanced generative model, our RH-Partial2Global method achieves an average performance improvement of over 3% compared to the Partial2Global baseline.\n\nIn summary, our proposed methodology offers a universal solution to this selection problem. We believe that studying prompt selection algorithms on these established benchmarks provides fundamental and generalizable insights into the mechanics of VICL. **This research path is complementary and, we argue, essential to the parallel progress being made in developing large-scale models.**"}, {"review_id": "4Z7HhWFnal", "reviewer": "Reviewer_rrVW", "summary": "This paper introduces a variant framework, called  RH-Partial2Global, based on a previous work Partial2Global for in-context example selection in Visual In-Context Learning (VICL). \nThe authors first challenge the default similarity-priority assumption in the Partial2Global framework that similarity between images can create better in-context examples. Then they point out that the random sampling strategy of candidate examples leads to incomplete coverage of pairwise relationships. To address the two key limitations in Partial2Global framework, the authors try to build a reliable and holistic in-context example selection from a statistical perspective. Specifically, they propose two strategies-a jackknife conformal prediction-based example selection strategy and a covering design-based sampling strategy in their work. Extensive experiments show that their proposed work performs better than the baseline Partial2Global framework across diverse vision tasks.", "questions": "1.\tFor Conformal Prediction, pre-defined confidence is necessary, but different settings will affect the final performance. The paper  does not seem to include an analysis of the effect of different confidence scores in CP. Or did I miss this part? \n2.\t247-248 lines, ‘... incurs negligible additional computational overhead ...’, the authors should provide the exact cost burden to illustrate?\n3.\tIn the paper, the baseline model Partial2Global is regarded as the state-of-the-art method. However there are other papers in the VICL area like: ‘Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning’, it has better performance than RH-Partial2Global method. Could you explain why you do not compare with the methods?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1.\tThe authors offer a new perspective on reliable prompt selection for VICL. In the paper, the work is the first work to introduce Conformal Prediction (CP) to the VICL, as far as the reviewer is aware. Based on CP, a jackknife selection strategy is proposed to construct a more trustworthy alternative set of in-context examples. \n2.\tThe authors incorporate a covering design-based sampling strategy within the consistency-awaking aggregator of Partial2Global, to have more holistic coverage of pairwise preferences and consequently with a more accurate global ranking. \n3.\tExtensive experiments on various datasets and tasks illustrate the performance and validity of the proposed methodology. To validate the validity of different strategies, comprehensive ablation studies were conducted, demonstrating the effectiveness of these strategies.\n\nWeaknesses:\n1.\tAlthough the paper proposed two strategies to improve the performance in VICL, the novelty is not enough.  The authors only use the jackknife CP to refine the construction of alternative sets and replace the original random sampling in Partial2Global with a covering design-based strategy.\n2.\tFor Conformal Prediction, pre-defined confidence is necessary, as different settings will affect the final performance. The paper does not include any analysis of the effect of different confidence scores in CP.", "was_revised": true, "final_justification": "The authors have addressed my concerns quite well and I am happy to increase the rating of their paper.", "rebuttal": "### **For Weakness 1: Regarding the novelty of our work**\n\nWe thank the reviewer for the feedback on the novelty of our work. We would like to respectfully clarify that the core novelty of our work lies not only in the individual strategies themselves, but also in our strong motivation from statistical perspective.\n\n**Our first contribution is to present the first rigorous and formal challenge to the ubiquitous similarity-priority assumption that underpins most current VICL methods.** We do this not by intuition, but by conducting a statistical hypothesis test that provides quantitative evidence of this assumption's shortcomings. To our knowledge, this is the first work to so rigorously diagnose this foundational issue. This strong, data-driven motivation is the bedrock of our paper. Our main experiments also demonstrate the effectiveness of the proposed jackknife conformal prediction-guided selection strategy targeted at this motivation.\n\n**In addition, our CP-based selection strategy can be identified as a universal and highly generalizable method.** please note that we also conducted extensive experiments on the VPR baseline to validate the universality of CP-guided selection strategy, achieving performance gains of **1.5%** to **3%** in our **Supplementary Material** **(Table 1)**. These results demonstrate that our CP-based selection strategy is not a narrow trick for one specific method but a **generalizable contribution** that can advance how the entire VICL field approaches candidate set construction.\n\n------\n\n### **For Weakness 2 and Question 1: Regarding the analysis of confidence level**\n\nWe sincerely thank the reviewer for this critical and highly constructive point. We agree that a deeper exploration of our method's sensitivity to the calibration set size, and its interaction with the confidence level $\\alpha$, is essential for understanding its practical applicability and limitations.\n\n**To demonstrate the robustness of our proposed strategy with respect to the confidence level $\\alpha$, we present the following ablation studies on VPR and Partial2Global.** We would like to draw your attention to the ablation study in the **Supplementary Material (Figure 1)**. This study, conducted on the VPR baseline, shows that our strategy consistently delivers performance improvements across a wide range of $\\alpha$ values (from 0.5 to 0.95). This result establishes the general robustness and effectiveness of our core approach. *(Note: We used VPR for this specific analysis because its architecture provides a cleaner and more direct assessment of $\\alpha$'s impact, allowing for $\\alpha$ more isolated analysis of the hyperparameter itself.)*\n\n**For comprehensive analysis, we conduct a corresponding ablation study on RH-Partial2Global**, evaluating its performance with $\\alpha$ values ranging from 0.96 to 0.80. The results are presented below.\n\n| $\\alpha$ | 1.00  | 0.94  | 0.93  |   0.92    | 0.91  |   0.90    | 0.89  | 0.88  | 0.87  |   0.86    | 0.85  | 0.84  | 0.83  |   0.82    | 0.81  | 0.80  |\n| :------: | :---: | :---: | :---: | :-------: | :---: | :-------: | :---: | :---: | :---: | :-------: | :---: | :---: | :---: | :-------: | :---: | :---: |\n|  Fold-0  | 38.81 | 39.19 | 39.35 |   39.47   | 39.02 |   39.11   | 39.44 | 39.10 | 39.03 |   39.24   | 39.25 | 39.08 | 39.12 | **39.48** | 39.32 | 39.36 |\n|  Fold-1  | 41.54 | 42.28 | 42.17 | **42.39** | 42.09 |   42.31   | 42.36 | 42.14 | 42.14 |   42.19   | 42.15 | 42.20 | 42.31 |   42.09   | 42.24 | 42.02 |\n|  Fold-2  | 37.25 | 38.25 | 38.11 |   38.25   | 38.12 | **38.28** | 37.86 | 37.99 | 38.08 |   38.24   | 38.06 | 37.99 | 38.05 |   37.93   | 38.01 | 38.09 |\n|  Fold-3  | 36.01 | 36.58 | 36.52 |   36.57   | 36.27 |   36.51   | 36.30 | 36.70 | 36.48 | **36.81** | 36.60 | 36.57 | 36.76 |   36.46   | 36.47 | 36.35 |\n\nThe results demonstrate that all tested $\\alpha$ values yield a performance improvement over the baseline (i.e. $\\alpha=1.00$).\n\n------\n\n### **For Question 2: Regarding the exact cost burden**\n\n**The additional computational cost of our method stems from a one-time, offline preprocessing step: the construction of the conformity score sequence.** This process, which involves running inference on the training set, is highly parallelizable and introduces a modest overhead, equivalent to approximately 5% of the baseline method, Partial2Global's total training time when measured on the same hardware.\n\n------\n\n### **For Question 3: Regarding the comparison with *Condensation***\n\nWe thank the reviewer for bringing the paper 'Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning' (*Condensation*) to our attention. This provides a valuable opportunity to clarify the fundamental distinction between two paradigms in VICL: **retrieval-based methods**, which operate under black-box assumptions, and **finetuning-based methods** that require white-box access. \n\n**On one hand, our  work, alongside VPR and Partial2Global, belongs to the retrieval-based paradigm and is designed to operate in a strict black-box setting.** These methods treat the core generative model as an opaque system, relying only on weak supervision signals such as final ranking scores. They require no knowledge of the model's internal architecture, weights, or intermediate features. This black-box formulation is notably more challenging but also more practical and general, especially as VICL increasingly involves large, proprietary foundation models accessible only via APIs. \n\n**On the other hand, the finetuning-based method like InMeMo [1] and the mentioned *Condensation* necessitates white-box (or at least grey-box) access.** They necessitate direct access to and manipulation of intermediate representations or even require backpropagation of gradients through the model, which constitutes a form of privileged information that is unavailable in a true black-box scenario. **Given that our method operates under a more constrained and challenging black-box assumption while *Condensation* leverages the advantages of a white-box setting, a direct performance comparison would be inequitable.**\n\n**Furthermore, the two paradigms are not mutually exclusive but potentially synergistic.** We note that *Condensation* also begins by constructing an initial candidate through similarity priority assumption (as illustrated in **its Section 3.1**). Our principled, CP-based selection strategy could therefore be employed as a more robust front-end to enhance this method, providing a higher-quality set of candidates for its condensation process.\n\n[1] Instruct Me More! Random Prompting for Visual In-Context Learning\n\n-----\n\n### **For Paper Formatting Concerns:**\n\nWe thank the reviewer for their valuable feedback regarding the formatting and clarity of our figures. For Figure 1, we appreciate the reviewer pointing out the inconsistent aspect ratios. This was an artifact of the model's preprocessing pipeline, as the MAE-VQGAN backbone requires all input images to be resized to a fixed 112x112 resolution for concatenation, which we depicted literally in the original figure. For Figure 2, we will increase the font size for all axes, labels, and legends to improve readability."}, {"review_id": "FB9bIcWhzJ", "reviewer": "Reviewer_jvC8", "summary": "The paper proposes RH-Partial2Global, a drop-in replacement for the widely used Partial2Global prompt-selection pipeline in Visual In-Context Learning (VICL). The authors first show that the common “pick the K most visually similar images” heuristic only weakly predicts prompt quality, and that the usual random shuffling of candidates leads to redundant, incomplete pairwise comparisons. To address these issues they introduce two training-free modules: (i) a reliable selection stage that uses jackknife conformal prediction to keep only those candidate prompts whose quality is statistically consistent with their similarity scores, and (ii) a holistic sampling stage that replaces random shuffles with a combinatorial covering design guaranteeing every image pair is compared at least once while minimising duplicates.", "questions": "1. Given the marginal improvement, can you report mean ± sd over several random seeds to substantiate the observed gains?\n2. Have you tried RH-Partial2Global on classification-only VICL or on video prompt selection?\n3. Would it benefit multi-object detection?", "limitations": "Except for the weaknesses mentioned above, this work's generality is still narrow; all tasks are image-level, no tests on video, multi-object detection, or classification-only settings.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Introduces conformal prediction and covering-design theory to VICL in a principled way. \n2. This method can be integrated into any Partial2Global-style method without requiring retraining of the backbone or ranker.\n3. The paper is well-written and provides algorithm pseudocode.\n\nWeaknesses:\n1. The biggest concern is that this work has modest absolute gains – improvements are <1%.\n2. This method is sensitive on dataset size, folds with fewer samples benefit less.", "was_revised": true, "final_justification": "Given the marginal improvement and dataset size sensitivity, I would keep my rating.", "rebuttal": "### **For Weakness 1: Regarding the performance gains**\n\nWe thank the reviewer for this excellent and critical question. We agree that a deeper analysis of the performance gains, particularly for the cases where they appear modest on the Partial2Global baseline, is both necessary and beneficial. Our analysis suggests that the magnitude of the performance gain is influenced by several factors.\n\n1. **We would like to highlight that our method's impact varies depending on the baseline it is applied to.** As detailed in the **Table 1 of Supplementary Material**, our strategy yields substantial improvements when integrated with other baselines, such as the VPR variants, where we observed gains ranging from **1.5% to 3%**. This demonstrates the general efficacy and high potential of our proposed method under different conditions.\n\n2. **Moreover, in our initial main experiments, we deliberately used a single, fixed confidence level ($\\alpha=0.85$) across all tasks.** This was to demonstrate the \"out-of-the-box\" utility and ease of use of our method without task-specific tuning. To test the hypothesis that gains could be larger with a tuned $\\alpha$, we conducted a new experiment as follows. \n\n   |    Fold-0     |    Fold-1     |    Fold-2     |    Fold-3     | Avg.  |\n   | :-----------: | :-----------: | :-----------: | :-----------: | :---: |\n   | $\\alpha=0.85$ | $\\alpha=0.85$ | $\\alpha=0.85$ | $\\alpha=0.85$ |  --   |\n   |     39.25     |     42.15     |     38.06     |     36.60     | 39.02 |\n   | $\\alpha=0.82$ | $\\alpha=0.92$ | $\\alpha=0.90$ | $\\alpha=0.86$ |  --   |\n   |     39.48     |     42.39     |     38.28     |     36.81     | 39.24 |\n\n   The results show that when $\\alpha$ is selected on a per-dataset basis, the performance gains on Partial2Global become more significant. \n\n3. **Furthermore, MAE-VQGAN, the common fundamental inpainting model used in current VICL methods like VPR and Partial2Global, is a little out-of-date.** Therefore, although RH-Partial2Global is highly optimized, it operates near the ceiling for MAE-VQGAN. \n\n   **To validate our opinion, we standardize the backbone architecture by replacing MAE−VQGAN with SegGPT [1], a more advanced VICL model in both the Partial2Global baseline and our proposed RH-Partial2Global framework.** Since the prompt selection mechanism in SegGPT involves randomly sampling images that share the same category label from the training set of PASCAL-5i as the prompt image (as illustrated in its Appendix A), we contend that this approach constitutes a form of information leakage, as it implicitly provides the model with class-level knowledge. Consequently, to create a stricter and more fair baseline, we utilize the single most similar image as the prompt and evaluate under our settings. This allows the evaluation to focus solely on the merits of our proposed strategies. The quantitative results for segmentation task are presented as follows.\n\n   |           Method           | Fold-0 | Fold-1 | Fold-2 | Fold-3 | Avg.  |\n   | :------------------------: | :----: | :----: | :----: | :----: | :---: |\n   |           SegGPT           | 64.35  | 66.79  | 58.91  | 65.68  | 63.93 |\n   |  Partial2Global (SegGPT)   | 66.52  | 71.55  | 62.48  | 68.74  | 67.32 |\n   | RH-Partial2Global (SegGPT) | 69.45  | 74.35  | 66.33  | 71.90  | 70.51 |\n\n   Notably, by integrating a more advanced generative model, our RH-Partial2Global achieves an average performance improvement of over **3%** over Partial2Global.\n\n   [1] SegGPT: Segmenting Everything In Context\n\n------\n\n### **For Weakness 2: Regarding the sensitivity to dataset size**\n\nWe agree with this observation and, in the spirit of transparency, had already identified and discussed this sensitivity to dataset size in the **Limitations** section of our original manuscript. \n\n**Regarding this phenomenon, we provide a principled analysis grounded in the theory of conformal prediction: The effectiveness of conformal prediction relies on the exchangeability of data, which is more robust with larger datasets.** With smaller sets (as in our Fold-0 and Fold-3), two issues may arise: (i) The empirical distribution of conformity scores becomes more coarse-grained, making the quantile estimation less precise. (ii) The set of scores is more susceptible to the influence of outliers or sampling noise, which can destabilize the calibration process. Recognizing the importance of this issue, we will conduct a more in-depth investigation into the interplay between dataset size, hyperparameter choice, and performance in our future research.\n\n**Nevertheless, we wish to emphasize that despite this inherent sensitivity in low-data regimes, our proposed selection strategy remains a powerful and generally applicable contribution.** The primary goal of our work is to introduce a more principled approach to candidate selection, and its broad effectiveness is demonstrated by the performance gains of both on VPR, Partial2Global and SegGPT. This indicates that while the magnitude of the benefit is moderated by dataset size, a known characteristic of conformal methods, the overall value and utility of our framework across different model architectures are robust. \n\n------\n\n### **For Question 2: Regarding the presentation of standard deviation**\n\nThe primary source of randomness in our experiments stems from the sampling step within the aggregation process of the Partial2Global baseline. To ensure the stability of our results, the performance metrics reported in our original manuscript were, in fact, already the **average of 10 independent runs**.\n\nWe have now calculated the standard deviation from these 10 runs. The results are now presented in the **mean ± standard deviation** format as follows. \n\n|      method       |     Fold-0      |     Fold-1      |     Fold-2      |     Fold-3      |\n| :---------------: | :-------------: | :-------------: | :-------------: | :-------------: |\n| RH-Partial2Global | 39.25$\\pm$0.097 | 42.15$\\pm$0.098 | 38.06$\\pm$0.100 | 39.25$\\pm$0.098 |\n\nThis tables show that the standard deviations are consistently small, confirming that the observed gains, while sometimes modest, are statistically stable and reproducible, rather than being an artifact of random variation. We believe this addition significantly strengthens the empirical evidence for our method's effectiveness.\n\n------\n\n### **For Question 2, 3 and Limitations: Regarding the generality of our work**\n\nWe thank the reviewer for these insightful suggestions regarding the application of our method to other important domains such as video, multi-object detection, and classification-only tasks. These are indeed exciting future directions for VICL research, and we appreciate the opportunity to clarify the scope of our current work and the generalizability of our proposed framework.\n\nThe primary reason for focusing on image-level tasks in this study is the set of established conventions and technical limitations of the prevailing VICL frameworks.\n\n1. **Backbone Model Constraints:** The standard foundational model used in this line of research, MAE-VQGAN, is architecturally designed for image synthesis and inpainting. which does not inherently support the processing of temporal data.\n2. **Lack of Established Task Formulations:** For tasks like classification-only VICL, there is currently no consensus or established framework in the literature for constructing the necessary \"example pairs\" (akin to the image-segmentation mask pairs used in our work). Defining what constitutes an effective visual prompt for a classification label is an open research problem in itself.\n\n**Despite these current constraints, we would like to emphasize that one of the core contributions of our paper, the CP-based prompt selection strategy, is conceptually modality-agnostic.** Our framework addresses the fundamental problems of how to best construct a reliable candidate set and how to select from it, a process that is independent of whether the input data is an image, a video frame, or another data type. Therefore, we strongly believe that our framework is forward-compatible. As more powerful and versatile foundational models emerge that are capable of video-based or multi-task in-context learning, our selection strategy can be seamlessly integrated to enhance their prompting efficiency and reliability. The principles of leveraging conformal prediction to manage uncertainty and build a reliable set remain directly applicable."}]

Author rebuttal:

Meta-review: This paper presents a well-motivated and methodologically sound enhancement to the Partial2Global framework for visual in-context learning prompt selection. The work is commended for its critical contribution in rigorously challenging a foundational yet largely untested assumption in the field—the "similarity-priority" heuristic. The authors provide quantitative evidence, via Spearman's rank correlation, that this intuitive assumption is often weak, which in itself is a valuable insight for the research community.

The proposed solutions are the paper's core strength. The use of jackknife conformal prediction to construct a statistically reliable candidate pool is a principled and data-driven approach to address the shortcomings of the similarity heuristic. Furthermore, replacing random sampling with a covering design strategy is a clever and robust method to ensure more comprehensive and non-redundant pairwise comparison coverage, directly tackling a key weakness in the prior art.

The experimental validation is thorough, demonstrating consistent and reproducible performance improvements across multiple diverse vision tasks (segmentation, detection, colorization) and metrics. A significant advantage is that these gains are achieved without any additional model training, highlighting the efficiency and practical utility of the proposed framework. The ablation studies are well-designed and effectively justify the contribution of each component.

While the absolute performance gains may be modest in some instances, the paper's conceptual rigor, methodological novelty, and consistent positive results across the board make it a strong contribution worthy of acceptance. It offers a valuable step towards more reliable and statistically grounded methods for in-context learning.
Decision: Accept (poster)