PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment
Authors: Tianci Luo
Jinpeng Wang
Shiyu Qin
Niu Lian
Yan Feng
Bin Chen
Chun Yuan
Shu-Tao Xia
Venue: ICLR 2026 Poster (2026)
Keywords: visual in-context learning
visual prompt
prompt fusion
locality-aware
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose the PromptHub paradigm, which enhances multi-prompt Visual In-Context Learning through locality-aware fusion, concentration, and alignment, thereby achieving superior effectiveness, transferability, and interpretability.

Abstract: Visual In-Context Learning (VICL) aims to complete vision tasks by imitating pixel demonstrations. Recent work Condenser pioneered prompt fusion that combines the advantages of various demonstrations, which shows a promising way to extend VICL. Unfortunately, the patch-wise fusion framework and model-agnostic supervision hinder the exploitation of informative cues, thereby limiting performance gains. To overcome this deficiency, we introduce PromptHub, a framework that holistically strengthens multi-prompting through locality-aware fusion, concentration and alignment. PromptHub exploits spatial priors to capture richer contextual information, employs complementary concentration, alignment, and prediction objectives to mutually guide training, and incorporates data augmentation to further reinforce supervision. Extensive experiments on three fundamental vision tasks demonstrate the superiority of PromptHub. Moreover, we validate its universality, transferability, and robustness across out-of-distribution settings, and various retrieval scenarios. This work establishes a reliable locality-aware paradigm for prompt fusion, moving beyond prior patch-wise approaches. Code is available at https://github.com/luotc-why/ICLR26-PromptHub.

Published as a conference paper at ICLR 2026
PROMPTHUB: ENHANCING MULTI-PROMPT VISUAL IN-CONTEXT LEARNING WITH LOCALITY-AWARE FUSION, CONCENTRATION AND ALIGNMENT
Tianci Luo!.", Jinpeng Wang2.*11 Shiyu Qin', Niu Lian",
Yan Feng", Bin Chen Chun Yuan Shu-Tao Xia
TTsinghua Shenzhen International Graduate School, Tsinghua University Harrii Institute of Technology, Shenzhen 3 Meituan, Beijing
itc250mail is s ssingua.... edu. en; iang com; chenbin2021 @hi t. edu. en; yuancêsz. twwignuunguu.nn edu. cn
ABSTRACT
Visual In-Context Learning (VICL) aims to complete vision tasks by imitating pixel demonstrations. Recent work (Wang et al. 2025) pioneered prompt fusion that combines the advantages of various demonstrations, which shows promis- ing way to extend VICL. Unfortunately, the patch-wise fusion framework and model-agnostic supervision hinder the exploitation of informative cues, thereby limiting performance gains. To overcome this deficiency, we introduce PromptHub, framework that holistically strengthens multi-proopping through locality- fusion, concentration and alignment. PromptHub exploits spatial priors to capture richer contextual information, employs complementary concentration, alignment, and prediction objectives to mutually guide training, and incorporates data augmen- tation to further reinforce supervision. Extensive experiments on three fundamental vision tasks demonstrate the superiority of PromptHub. Moreover, we validate its universality, transferability, and robustness across out-of-distribution settings, and various retrieval scenarios. This work establishes reliable locality-aware paradigm for prompt fusion, moving beyond prior patch- wise approaches. Code is available at lttttts::/// githhb..thuugg.og com/ luote why/ ICLR26 HomptHub
INTRODUCTION
Foundation models like GPT (Brown et al. 2020). Llama (Touvron et al. 2023). Gemini (Team et al.
2023) and Flamingo (Alayrac
2022) have demonstrated the emerging ability of demonstration-
based prompt learming, aka In In-Context Learning (ICL) (Dong et al. 2024 Zheng et al. 2023 Yang
al. 2023), which further facilitates their versatility various tasks. The basic idea of ICI. (Hendel
et al.
2023 Wei et al.] [023] 2022 Jiang et al., 2024) is to prompt models with some demonstrative
input-output pairs in addition the query input, which can enhance the answer robustness. The
reliablity of ICL has been thoroughly validated (Shin et 2002 Yoo et al.) 2022 Dai al. 2023
Von Oswald et al. 2023). Recently, Visual ICL (VICL) (Bar et al. 2022] Wang et al. 2023b have
also become a popular topic, where pixel-space in-painting is the native paradigm.
Choosing appropriate prompts is critical in VICL. Many recent works (Zhang et al. 2023 Sun et al.,
2025 Xu et al. 2024:) Zhu et al. (2025) focused optimizing retrievers to select better suited prompts, and Zhang et al. (2024) incorporated visual prompt tuning to enhance the robustness of VICI. Nature Language Processing (NLP) literature (Shi al. 2022 Gao al.. 2024) suggests multiple prompts can enhance ICL with mitigated bias and richer context, offering insights essential for advancing VICL. Yet visual backbones like MAE-VQGAN typically restrict inputs a single prompt, rendering multi-prompting non-trivial. In practice, there are two heuristic strategies for extending single- prompting to multi- multi-prompting namely downscaling (Wang al. 2023a and ensemble (Sun et
"Equal contribution.
1 "orresponniig authors.

---

Published as a conference paper at ICLR 2026
(o) Multi prompttiopiigggg via CONDENSER
(b) Multi soautt-prompting via PromptHuu (locality fusion - utilizotion- prediction chain) Prompt pairs (Data aug)
PremptHub Vs. CONDENSER Segmentation (mIou)
Prompt pairs (Top N)
Query image
Query image
+2 +2.5%
(Effective spatial global fusion)
Cneffective Tusion]
-
Patch-wiee sitentioon
Patch-wise attention list
Localiiy-enaance
cross sttention
Locolity- aftection mop
K=1
K-16
Target convos
Target convas
[Fused promot feature alanment)
Detection (mZgl)
-2.1%
+..0%
(Model agnossict
K+]
K+16
Mask
Discrete tokens
Colorization (MSE)
(Reinforce fused utilization
5.19
Continuous
Continuous
7.2%
(Label prédiction)
K-1
K-16
CONDENSER
PromptHub
prodiction)
Figure 1: (a) CONDENSER performs patch-wise fusion to fuse composite prompt, while leveraging model-agnostic supervision signals at the input level. (b) PromptHub transcends CONDENSER by enforcing locality-aware chain that unifies fusion-utilization. iitiiaatioo-reeicttion.. It aligns spatial priors into coherent prompt representations, reinforces the backbone's 's concentration on fused cues, and integrates label prediction to maintain the integrity of VICL pipeline. (c) Comparison CONDENSER and PromptHub across three tasks under both
and multi- prompting configurations.
2025) Building upon this, CONDENSER (Wang al. 2025) was the first adopt prompt fusion (Liu
et al. 2023b), integrating useful information from multiple prompts into a fused prompt, as illustrated Figure [). However, its patch-wise fusion strategy results in substantial underuse of valuable cues, while model-agnostic supervision remains insufficient. Moreover, discrepancies between fused prompt and query pair may compels backbone distrust fused representation, falling back on its own capacity for inference. This is precisely the situation we aim to avoid.
We break these limits by proposing PromptHub, which aims (i) integrate precise knowledge from diverse prompts. (ii) mitigate fused prompt's 's discrepancies to encourage the backbone's 's effective trust and reliance on it, and (iii) ultimately yield superior VICL predictions. To achieve this, we propose locality- aware fusion framework together with three cooperative learning objectives, as illustrated in Figure (h). Specifically, we introduce a locality prior that applies spatially decaying weights radiating from the current patch, thereby enhancing accurate feature extraction. This design allows the fusion process to retain a global receptive field while alleviating the adverse effects of border noise. During the optimization of PromptHub, we design three complementary objectives: (i) end-to-end semantic integrity loss to promote high-quality prompt fusion by aligning fused exemplars with query semantics, semantically closer prompt generally benefit VICL; (fi) utilization loss that mitigates discrepancies between the fused prompt and the query pair, thereby promoting the backbone's 's trust and reliance the fused representation for imitation learning: and (iii) label prediction loss, retained from CONDENSER, which serves as the base supervision to preserve VICL's contextual prediction behavior. Additionally, we preliminarily explored VICL-oriented data augmentation strategies to enhance the robustness of PromptHub. These designs realize chain-wide enhancements for VICL,
We evaluate PromptHub on segmentation, detection, and colorization. As shown in Figure C),, extensive experiments demonstrated its superiority to state-of of of-the-art baselines. We also demonstrate PromptHub's 'ss promising resource efficiency, transferability and robustness to various prompt retrieval strategies including random selection. Comprehensive ablations on diverse learning objectives, data augmentation techniques, and locality fusion, confirming the success of our paradigm design. We further visualize the fused prompts, which validates the reliablity of PromptHub. These findings strongly support the efficacy of PromptHub, highlighting the significance of our approach in VICL.
To sum up, make the following contributions.
We introduce locality fusion strategy that balances spatial locality and receptive field, enabling more comprehensive extraction of effective information.
2

---

Published as a conference paper at ICLR 2026
We propose three complementary learning objectives that collaboratively enhance prompt fusion quality, strengthen prompt concentration, and improve contextual prediction, further reinforced with VICL-specific data augmentation.
Extensive experiments show PromptHub's efficacy beyond state-offle oftte-of-thh-ar art techniques. Promis- ing results also suggest that transferable across domains and robust to prompt retrieval, establishing reliable competitive solution in VICI..
RELATED WORKS
2.1
LARGE VISION MODELS
The field of computer vision has witnessed substantial advancements, driven by abundant foundational
models (Chang et al., 2022 Wang al. 2023c Oorloff et al. 2025 Qiu et al. [2255] Yu et al. (026). LVM (Bai et al. 2024), an auto-regressive generative model, effectively converted visual information
into language- like visual sentences and improved understanding capabilities. MAE and Point-MAE (He et al. 2022 Pang et al. 2022). , utilizing a reconstruction strategy, established unified visual architectures across various downstream tasks in 2D and 3D domains. The ability for ICL has also been demonstrated within foundation models, as researchers employed specialized training methodologies (Bar et al., 2022 Fang et al. 2024 Wang et al. 2023b Wei et 2025| Yue et al.) 2024, Yu et al., 2025 Yue et al. (023) to endow these models with superior in-context learning capabilities, thus providing a robust foundation for the domain of Visual ICL (VICL).
2.2 VISUAL IN-CONTEXT LEARNING VIA IN-PAINTING
MAE-VQGAN (Bar et al.
2022) and Painter WWng et al. 2023a serves as crucial in-painting
backbones for VICL, with copious works building upon enhancing this framework. Existing work (Zhang et al. 2023 Sun et al. 2025 Xu et al., 2024) primarily focuses on retrieval to obtain better prompt. Sun et al. (2025) studied prompt spatial arrangement, testing eight configurations and reporting improved results through voting. Zhang et al. (2024) pioneered visual prompt tuning PPeiiffer et al. 2020 Hu et al., 2021 Liu et al., 2023a Bahng et al. 2022). adding a noise border to prompts. PANICL (Zhang et al. 2025) employs training-free k-ncarest-neighbor fusion integrates multiple prompts to alleviate the bias inherent single prompt. PICO (Tiang el al. 2025) reformulates personalized vision problem under the VICL paradigm and exhibits clear advantages. Hojel et al. (2024) focused on identifying task vector activates backbone to optimize VICL process. CONDENSER (Wang et al. 2025) leveraged prompt composition Li et al. (2024) aggregate informative cues from multiple prompts. However, the patch-wise information aggregation strategy in CONDENSER (Wang et al., (2025) exhibits inherent limitations, and its supervision over exemplar quality remains insufficiently comprehensive. Motivated by these, we propose a locality fusion scheme coupled with three cooperative objectives, establishing a more reliable paradigm for prompt fusion.
3 METHOD: PROMPTHUB
3.1 PROBLEM FORMULATION AND METHOD OVERVIEW
Given a prompt database, D [P)) where each prompt comprises an image-label pair. The pixel-level retriever R identifies top-N similar prompt pairs P= (P. = (X,,, Y. for a given query image X, E Hx Following previous settings, patch size of 16 and feature dimension of D, as the backbone. Under general setting, the closest pair Pi (X1, Yi) IR #x2W x3 provides the prompt. We concatenate prompt P1 with the query image
adopt MAE-VQGAN, configured with
Xi Yi
X, to construct the canvas S
[M]] where [M] denotes the mask need to be recovered.
X,
The backbone output the masked location corresponds the X,, predicted label Y
In our framework, N prompt pairs P and query image Xg are processed by the query-adaptive Hub module, producing fused features Fx,: Fy, E R
H x 1h xDD Fxy: Fy, and query image
3

---

Published as a conference paper at ICLR 2026
Training stage Prompt pair Query pair
Inference stage Discrete tokens
Discrete tokens
VICL answer
VQGAN Encoder
VQGAN Decoder
VQ
Ohp
Retrieve prompt pairs
OUT
MAE Encoder
PromptHub
Mask
Data
base
Fused in- in-context sample In the embedding space
Query image
Continuous tokens
Figure 2: The training and inference framework PromptHub based on
Fx, Fy, Fx, passed through MAE-VQGAN, excluding the patch embedding, generate the VICL answer.
features Fx,: mask features FIM] are concatenated into the canvas Sj =
Then Sy is
F[M]
3.2 PROMPTHUB MODULE DESIGN
To expand the receptive field during fusion while mitigating the impact of boundary noise, we employ locality-enhanced prompt fusion strategy. PromptHub locally fuse into unified prompt pair (Fx,: Fy,) ) in the embedding space. The workflow is shown in Figure
We first process images (Xq: P) by embedding layer yield Ex,; EXINN BY,.. E IR x xD
Subsequently, we deploy a self-attention transformation SA(-) to align the query and prompts to similar patterns, thereby generating the resultant features Fx,, Fx..N. FYi..-
Thereafter, we use aueryvad from prompt pairs features (Fx..:: Fr..x), which achieves the fused exemplar features (Fx, Fyy)-
cross- attention to extract spatial information
We define the locality prior as a probability distribution controlled by the hyper-parameter a. To effectively represent this locality distribution, we employ either Gaussian prior or Laplacian prior, with the choice governed by hyperparameter:
exp
(z
- (y-
20
Locality prior Gaussian prior
(h, w. v)
(I)
+yy w)
exp
Locality prior = Laplacian prior
For each query image token Fx, [h, w]. it has a specific locality matrix V centered at (h, w):
wi w. 1,
(h, w. 1, 16
V 1r.
1b w. H 1)
(h, w, H W 1
During the VICL inference phase, matched query label Y, is available for constructing fused prompt label Fy,- However, the specific correspondence still exists between prompt images X and prompt labels YI.N. Therefore, share the prompt images features Fx,N as the key in the attention mechanism, compute the generalized attention scores, and subsequently perform localized weighting to procure locality-enhance attention weights A E N 4h x w. for Fx, [h, w]:
(Fx, w| x We) x (Fxix x WK)
A softmax
T h,w
4

---

Published as a conference paper at ICLR 2026
PromptHub pix
the embeddiing space
Query image
Ird onence SToge
N prompt pairs
Retrieve top -gg-Neen most simlar prompt
attention
Fused image
patch embedding
Self=
Training stage (w/ data oug)
Step 2: retriees Top-N simitr prompi pairs
MAE
Fused lobel
P
Jife
Locality-enhancee
Data bade
Query image
1
cross attention
query pairs
POPDOm
PromptHub module design. N prompt pairs and query image are embedded into the MAE patch space, where locality-enhanee fusion integrates spatially cues into a fused prompt aligned with query's ss informative content.
Figure 3:
Figure 4:
Data augment of
PromptHub. In training, the top-N pairs randomly substituted with either query pairs or random pairs.
which denotes element-wise multiplication, x denotes matrix multiplication, and Wo, WK DDD represent the projection layers for mapping Q and K in the attention mechanism.
Ultimately, we multiply the locality-enhancee attention weights with the features Fx,:N and FYN. through linearly transformed to obtain the fused prompt pair features Fx,, Fy, E R f 1h x.D. Wyx, Wvr &RPx denote linear layers in attention mechanism for image and label, respectively.
Fx, [h, w) = A x (Fxux x Wvx), Fy,lh, w) = AA x (PYn. Wyr). (4)
3.3 LEARNING OBJECTIVES
We introduce three complementary learning objectives guide fusion module's training, collectively strengthening ffastoon-uilization-peddiiiiiiii closed-loop for robust VICL, as illustrated in Figure
(i) Ensuring label prediction performance. Following CONDENSER and InMeMo, also adopt a label prediction loss as the fundamental objective to preserve VICL's contextual prediction behavior. Without this base supervision, parameterized VICL paradigms cannot function properly. Upon deriving the fused in-context sample Sy, we propagate it through the MAE encoder, generating
T Ty, TX, [M]]
canvas of continuous tokens
R x
These tokens are calibrated during
pretraining to correspond with the VQGAN codebook space. Simultaneously. We construct the target [,, Yg g
canvas Sg
E R22x22xx x3 by integrating the query pair as a prompt pair and process
X
I
I (
it through the VQGAN encoder, obtaining the corresponding discrete tokens
I
I d(2))
from the codebook. Here, Ne denotes the size the codebook space, with N. D. Td(11) and Yy represent the discrete tokens output prompt, while Te(2) and T correspond to the discrete tokens output as query. To optimize the label prediction results, align
{1, 2,,2.... No)
the bottom-right portion Tir.. which will be reconstructed by the VQGAN decoder, with the target Tel(2) using a cross-entropy loss. Here, L, denotes the loss function for label prediction:
Cp (E 11 (I ()) log TiM)
h, w. T, T2)(2)
(5)
(ii) Fused-promp feature alignment. The backbone tends to produce accurate predictions when exposed to the same prompt as the query. We employ cross-entropy alignment between the continu- ous tokens derived from the fused prompt pair (Tx TT,) and the discrete tokens corresponding to
the query pair prompt (Ix,) (ud(1)) make fused prompt pair closely approximate the query pair. The semantic integrity loss for improved fusion is denoted as L.-
(log Tx, [h, w, T To((1) + log Tú, [b, w. T
C. = ((1
(6)
5

---

Published as a conference paper at ICLR 2026
(iii) Enhance fused prompt utilization. Owing to discrepancy between fused prompt and query pair, backbone may regard useful prompt as unreliable and instead rely on its own capacity. We employ cosine- similarity loss Lus designed to reduce the dissimilarity between query pair (T and fused prompt (T& ,,,,, thereby enhancing the backbone's 's utilization of fused prompt.
Lu (B( M. x(1 I)D) cos (xx, [t, w), ,2x, [h, wil + cos (TY, (b, wl, Ton,TTnnlle wi))
(7)
We adopt À and y balance diffenent losses. Let 0 denote the parameters of PromptHub. The ultimate synergistic optimization objective is formulated as: min Cp + AL, + yhC..
(8)
3.4 RETRIEVE SCHEME FOR DATA AUGMENT
In the inference phase, we consistently retrieve the top-N most similar prompt pairs P = {Pr] J from the database D, utilizing the most relevant raw prompt pairs for improved VICL.
During training, we employ a data augmentation strategy to enhance two regularization objectives' effect. Based on the retrieved top-N prompt pairs P = {P, J might replace some prompt pairs P, with either query pairs P, = (X,, Yg) or randomly retrieved pairs Pr,, as shown in Figure 4
(i) Substitute with query pair better utilize the fused prompt. Under typical settings, defining prompt pair P1 as query pair Pg generally yields minimal discrepancy. To this end, we replace current prompt pairs P,, with query pairs P, with probability Py. This substitution establishes purified learning objective that minimizes discrepancy as much as possible, hence enhancing L...
(ii) Substitute with random pair to enhance PromptHub's robustness. With probability pr. we substitute prompt pair P, with a randomly retrieved pair Pr. introducing a controlled level of noise that enhances C. and PromptHub's ' stability. This technique guarantees when high-quality prompts are unavailable during inference, PromptHub retains its capacity to achieve robust VICL results.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Downstream Tasks and Datasets. To ensure a fair comparison, we employ three well-established tasks foreground segmentation, single-object detection, and colorization along with their associated datasets, within the domain of VICI.. For foreground segmentation, we employ Pascal-5 (Shaban ctt al. 2017), which consists of four folds, with each fold containing data from five different classes. We conduct experiments across all folds and analyze the results by presenting the mean intersection over union (mloU) for each fold. In the case of single-object detection, we utilize the Pascal VOC2012 (Everingham et al. 2015) dataset, also employing mloU the evaluation metric. For the coloring task, we randomly select 50,000 images from the ImageNet-1K ILSVRC2012 (Russakovsky al. 2015) training set, with 50 images chosen from each of the 1,000 classes form the label portion of our training set. The 50,000 images from the validation set of ImageNet-IK ILSVRC2012 are used as the label portion of our test set. We convert training set and test set label portion to grayscale images, which served as the input queries. We use MSE as the evaluation metric.
Implementation Details. We adopt MAE-VQGAN (Bar al. 2022) as the backbone architecture and utilize Prompt-SelF's (Sun et al. 2025) pixel-level retriever for prompt retrieval. During training. we use the training set as the database for prompt pairs while also employing the training set as the query. In the testing phase, the validation set serves the query collection, while the training set acts as the database. The input image resolution to the model is 224 224, with each sub-image having a resolution of 112 112. We utilized Gaussian prior the default locality prior.
We employed SGD optimizer with a learning rate initialized at 0.04, which decays according to cosine annealing warm restarts scheduler. For segmentation and detection tasks, training is performed for 100 epochs, while coloring task requires 10 epochs. The corresponding values for foreground segmentation, object detection, and colorization tasks are 0.65, 0.5, and 2.5, is set to 0.5, and y is set to 0.2. The experiments were performed on single 80G A A100 GPUs with a batch size of 16.
Training Configurations.
respectively.

---

Published as a conference paper at ICLR 2026
Table 1: PromptHub performance is compared with different baselines in three downstream tasks foreground segmentation (Seg.), single-object detection (Det.), and image colorization Co The results for N = 1, 16, representing the cases with 1 and 16 prompts respectively, are listed separately.
The highest results are denoted in bold, while the suboptimal results are indicated in italies.
Seg. (mloU f)
Model
Det. (mloU T)
Col. (MSE J)
Fold-0 Fold-1
Fold-2
Fold-3
Mean
Zero-Shot
Random (Bar ct al 2022
28.66
30.21
27.81
23.55
27.56
25.45
0.67
UnsupPR Zhhang et
2023
34.75
35.92
32.41
31.16
33,56
26.84
0.63
Prompt- SolF (Sun
35.69
38.25
35.86
33.37
35.79
28.08
0.63
Retriever Truining
SupPR ZZhnn et al.
2023
37.08
38.43
34.40
32.32
35.56
28.22
0.63
Gaotaal (Xu of al. 2024
38.81
41.54
37.25
36.01
38.40
30.66
0.58
PEFT
InMeMo Zhang et al
2024
41.65
47.68
42.43
40.80
43.14
43.21
Task Vectors
VTV (Hojel et al. 2024
38.00
37.00
33.00
32.00
33.50
Prompt Fusion
CONDENSERN=1 Wang al.
42.13
50.31
42.20
41.90
44.14
43.22
0.560
CONDENSER N-16 Waang 2025
45.53
52.06
44.33
44.58
46.63
44.64
0.539
PromptHub - (Ours)
43.26
50.75
43.83
42.82
45.17
44.51
0.533
PromptHub 11 (Ours)
45,93
53,12
45,44
46.74
47.81
45,59
0.503
4.2
COMPARISON WITH STATE-OF-THE-ARTSS
Baselines. We compare our method against comprehensive state-of-the-ae art approaches built on the MAE-VQGAN framework. Our competitors categorized into four groups: (1) Zero-shot methods, including MAE-VQGAN (Bar et al. 2022) and UnsupPR (Zhang et al. 2023 and Prompt-SelF (Sun) al. 2025), which do not require additional retriever training: (2) Methods that necessitate retriever training, such as SupPR (Zhang et al., (023) and Partial2Global (Xu et al. 2024); (3) Approach
that leverages prompt tuning, exemplified by InMeMo (Zhang et al. 2024); (4) Method of finding
and utilizing the task vector VTV (Hojel et al. 2024). (5) Method that employs prompt fusion,
CONDENSER (Wang et al., 2025). to enable multi-prompt VICL, with comparisons reported under both single-prompting and multi multi-prompting settings.
(i) Performance on Standard Tasks. Table demonstrates that PromptHub achieves consistent improvements across all tasks under both single-prompt and multi-prompt settings. In single-prompl scenario, PromptHub surpasses CONDENSER by 2.3%, 3.0%, and 5. 1% on segmentation, detection, and colorization, respectively. Under multi- prompt scenario, it further attains gains of 2.5%. 2.1 1,,, and 7.2% on same tasks. PromptHub's output visualization is discussed further in the appendix.
Domain Adaption Task. In real-world applications, the data for infer- ence may undergo domain adaptation compared to the training data. Thus, testing the trans- ferability of different VICL schemes crucial. We trained the PromptHub on the COCO-5' Lin al., 2014) using the same settings as previ- ous works (Wang et al. 2025; Sun et al. 2025 Zhang et al. 2024), and evaluate on the Pascal-5'. As shown in Table PromptHub
(ii) Performance
Table 2: Transferability evaluation. We train models on COCO-5 and test on Pascal-5
Fold-O
Fold-1
Fold-2
Mean
Prompi-SelF
40.13
14
37.84
38.52
39.66
InMeMo
CONDENSERY_
40.39
44.54
40.23
36.33
40.37
CoNDENSERy_.1
40.37
41.85
41.03
35.84
40.52
PromptHoab
40.36
45:24
40:13
37.97
47,00
PromptHabx--s
demonstrates substantially larger improvements than other baselines, outperforming CON- DENSER by 4.1% in the multi-prompt setting, highlighting the strong transferability.
Segmentation
Detection
Colorization
46-
no dooes
nou
0.52
(iii) Performance under the multi prompin scenario. To validate the scalability of PromptHub, we compare it with CONDENSER under various N, specifically 1, 2, 4, 8. 16, and 32. In addition, report results under down- wwwn--smmllnngg--a 22pplinng=22,222 2 (Zhang et al. 2023) and answer-level (Sun et al. 2025 The
19 20.0
0.56
Promet Number
Promot Number
Promat Number
OPromptHub & Condenser
Down sampingg fusion
Arsworr fusior
Figure 5: Performance comparison with baselines in multi multi-prompt VICI. scenario.
7

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study of PromptHub. The best are marked in bold and second-best in italic.
Seg, (mloU f)
Model
Det. (mIoU f)
Fold-0
Fold-1
Fold-2
Fold-3
Mean
(0)
PromptHubg-1
43.26
50.75
43.83
42.82
45.17
44.51
(1)
PrompiHubx=
45.93
53.12
45.44
46.74
47.81
45.59
Effectiveness of Learning Objectives
(2)
w/o L. N=1
42.71
51.14
42.78
42.41
44.76
43.45
(3) w/o C.
45.54
52.25
44.59
44,47
46.71
44,83
(4)
w/o L.
42.23
50.52
42.29
42.16
44.30
43.12
(5)
w/o L. N=16
44.72
51.77
43.57
43,30
45.84
44.27
(6)
w/o Ep X=1
8.51
10.13
9.46
8,33
9.11
13.23
(7)
wlo Ep N=16
9.41
13.44
12.29
10.62
11.44
12.87
Effectiveness of Enhancy Fusion
(8) w/ Laplacian priorx-1
43.74
50.93
43.51
43.05
45.31
43.93
(9) w/ Laplacian P1iorx-16
46,42
52.87
45.45
46.16
47.72
45.47
(10) Global Fusionx=:
41.77
49.04
42.69
40.73
43.55
41.86
(11) Global Fusion
41.91
50.45
43.76
42.43
44.64
42.49
(12) Convolution- Based Fusiony=1
42.56
50.15
42.79
42.52
44.51
43.83
(13) Convolution- Based Fusiony
45.28
$1.68
45.34
45.51
46.95
45.07
Effectiveness of Data Augment Technique
(14) w/o Data Augmenty-11
43.11
51.22
43.17
42.34
44.96
43.52
(15)
w/o Data 1ugmenty=16
45.84
52.01
44.83
45.60
47.07
45.06
Prompt-SelF
Condenser H-16
IoU-16.84
loU-29.16
loU-15.99
loU-47.81
loU-26.11
PromptHub 8-16
IoU=40.94
loU=89.58
1oU-89.93
loU=77.62
loU=51.67
loU
loU=61.20
loU=50.22
Ground Truth
Figure The visualization of the fused prompt pair after passing through the VQGAN decoder.
experimental results demonstrate our approach not only improves performance as N increases, but also consistently surpasses other baselines by large margin, as shown Figure
4.3 MODEL ANALYSIS
For comprehensive ablation study, we designed several variants, as summarized in Table 3 where Variants (0) (1) correspond the canonical configurations.
(i) Effectiveness of Learning Objectives. To comprehensively evaluate the contributions of each learning objective, we conducted an ablation analysis by individually removing the three objectives. The experimental results demonstrate mutually complementary, and omitting any of them leads to performance degradation multi-promph VICL. In particular, the primary objective, label prediction Cp. is indispensable for preserving VICL's contextual prediction behavior; without it, the trainin-bbssd VICL paradigm with additional parameters cannot function effectively. Meanwhile, L, and L. act erucial regularization terms, ensuring fused exemplars' quality and the backbone's 's effective utilization. The absence of either damages the pipeline in VICL and results in mediocre performance.
-uliizatiiooneeeeictiono prediction" " objectives are
(ii) Effectiveness of Enhanced Prompt Fusion. We compare locality-enhanced fusion with global fusion, wise fusion (CONDENSER (Wagg et al. 2025)). and convolution-hasel
8

---

Published as a conference paper at ICLR 2026
fusion, where the latter replaces the spatial prior with convolutional transformations. Notably, locality-enhanced fusion can be viewed as a higher-level framework, within which global fusion and patch-wise fusion emerge two complementary instantiations, corresponding to larger and smaller values of the locality parameter o, respectively. As shown in Table both types of locality priors achieve superior performance. Upon observation, maintaining an appropriate balance between global receptive fields and spatial locality proves essential. The locality accords with the fusion principle that enriches information capture while mitigating long-range noise.
(iii) Effectiveness of Data Augment Technique. We conducted experiments under scenarios without data augmentation, only utilizing the top-N prompt pairs for fusion during training, as illustrated in Variants (14) - (15). The results indicate that removing data augmentation diminishes the performance of PromptHub VICI. tasks, confirming the effectiveness of data augmentation. better reinforce fused prompt utilization and enhances noise resistance.
(iv) Impact on Different Retrievers. Explor- ing better prompt retrieval and investigating multi-
Random
UnsupPR
SupPR
IP Pwe-Level
53
prompt fusion are two orthogonal research directions,
50
while the fusion plugin can be udapted to different re- trievers. We investigated performance of PromptHub using different retrievers,
44
41
presented in Figure We evaluated four types of retrievers: random se- lection (Bar et al.] 2022). UnsupPR (Zhang al. 2023). SupPR (Zhang et al. 2023) and Pixel-Level retriever (Sun
38
Fold
(a)
Rendom
UroupPR
SupPR
Pixel-Level
53
2025). Experimental results
demonstrate PromptHub is more effective than CON- DENSER across all retrieval schemes, further high- lighting its generalizability. Additionally, the perfor- mance of our method influenced by the choice of re- triever; pixel-level retrievers consistently deliver best results, underscoring the alignment between pixel- level retrieval and locality-aware design philosophy.
47
44
41
38
Fold-0
Fold-1
Fold-2
Fold-3
Mean
PromptHub
ZZ
Condenser
Figure 7: Comparison of PromptHub and CONDENSER across different retrieval.
(v) Transferability on Unseen Tasks. We evaluate cross-task transferability by train- ing all models solely segmentation (Pascal-5, four folds) and directly testing them on detection (Pascal VOC 2012) without any fine-tuning. We compare PromptHub with the CONDENSER baseline using their released checkpoints, and report results in Table 4 PromptHub consistently surpasses Con- denser in this challenging unseen-task set- ting. With N = 16, PromptHub achieves +1.83% mIoU gain, indicating that our locality- aware fusion captures more robust and transferable visual cues than the patch- wise fusion used in Condenser. We note that although the overall performance is strong, the training process is not fully task- agnostic. Since the model is trained to re- construct segmentation masks, domain gap naturally emerges when transferring bounding box detection, which leads to a certain degree of performance drop.
Table 4: Transferability experiment (unseen task) where both CONDENSER and PromptHub are trained on seg- mentation and evaluated on detection.
Det. (mloU 1)
Method
Fold-0
Fold-1
Fold-2
Fold-3
Mean
Condenser
38.15
35.70
35.49
30.31
34.91
Condenserx
41.25
36.66
37.86
39.02
38.70
PromptHubx-1
41.59
36.25
37.71
32.15
36.93
PrompiHuhx-ze
43.40
38.55
39.52
40.66
40.53
Spatial misalignment between prompts and queries may negatively affect prompt fusion performance. To evaluate performance under position shifts, conducted periment where query pairs were horizon- tally flipped and retrained to simulate se- vere spatial misalignment between the re- trieved prompts and the query image. We
(vi) Performance Evaluation under Spatial Misalignment.
Table 5: Comparison of standard and perturbed mloU under spatial misalignment, along with the correspond- ing performance drops.
Method
Standard mlouU
Perturbed mloU
Performance Drop
compared the performance drop of CON- DENSER and PromptHub under the per- turbed conditions in Table 5 PromptHub substantially more robust to spatial misalignment than CONDENSER. Its locality-aware fusion
Condensery.1
44.14
42.36
-1.78
Condensery-jn
PrompiHubb-s
45.17
44.01
-1.16
PrompiHuby--s
47.11
47.15
-0.66
mitigates the sensitivity positional shifts that affects CONDENSER'S patch-wis wise fusion. In addition,

---

Published as a conference paper at ICLR 2026
increasing number of prompts N further reduces misalignment effects by improving chance of encountering better-aligned prompt pairs.
4.4 DISCUSSION: WHAT DOES PROMPTHUB LEARN?
In Figure 6 we present visualizations of Prompt-SelFF well as fusion samples reconstructed through the VQGAN decoder for CONDENSERN=IN and PromptHubx Given that this visu- alization relies on reconstructed outputs, some bias may be inevitably introduced. We observe that in Prompt-SelF, label prediction often tends to be highly similar to the retrieved prompt label, leading to poor performance when the retrieved label show little similarity ground-truth answer. The fusion results of CONDENSER appear as noisy -ldck-and-white patterns, which may be attributed to its model-agnostic fea- ture matching and patch-wise attention that fail to generate smooth representations, of- fering only heuristic contributions to perfor- mance. In contrast, the fused prompts pro- duced by PromptHub exhibit significantly better visual quality, with fused prompts showing high similarity to the query pairs and smooth textures, thereby conftrming the advantages of the locality-aware design and offering
Table 6: Comparison the mloU between the fused prompt labels and the query labels across methods to evaluate semantic alignment.
Seg. (mloU f)
Method
Fold-0
Fold-1
Fold-2
Fold-3
Mean
Condenserx
18.93
29.73
24.26
27.94
25.22
Condenserx-16
14.27
24.56
18.85
20.56
19.56
PrumptHubbx=
21.25
37.61
35.01
29.79
30.92
PrompiHHubx-1 25.22
43.22
36,07
30.41
33.73
more reliable and trustworthy solution for prompt fusion fusion in VICL. Furthermore, we quantitatively compare the mloU between fused prompt labels and query labels for CONDENSER and PromptHub, as shown in Table 6 PromptHuby
achieves a 72% higher similarity to the ground truth compared with CONDENSER N 16. demonstrating PromptHub produces higher-quality and more semantically coherent fused prompts. Although fused prompts may exhibit a gap from realistic images due to lack of fidelity constraints in decoding stage, our primary goal to guide VICL. inference rather than to generate photorealistic images.
5 CONCLUSIONS
In this work, we introduced PromptHub, a interpretability paradigm realizes the chain-wide en- hancements "locality suston-ulliaatooo-preditioo for multi-prompt VICL. PromptHub balances spatial locality with global receptive fields, supervises the quality of fused samples, and enhances the backbone's 's utilization integrated prompts. Extensive experiments across diverse tasks demonstrate clear improvements over previous methods. Furthermore, PromptHub's superior transferability, robustness and generalizability further highlight its potential for extensive implementation in diverse scenarios. We finally visualize the fused prompts, the results outperform patch-wise scheme and provide stronger interpretability for prompt fusion methods.
Acknowledgments. We sincerely thank the anonymous reviewers and chairs for their efforts and constructive suggestions, which have greatly helped us improve the manuscript. This work is supported in part by the National Key R&D Program of China 2022Y the National Natural Science Foundation of China under grants 624B2088, 62576122, 62301 189, 62571 1 298, and in part by the SSTIC Grants 3JZD20030922111100000 KJZDD20230923 and GJHZ20240218 3604008.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-237356 2022.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting scale models. arXiv preprint axXiv:2203.17774 2022.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models.
10

---

Published as a conference paper at ICLR 2026
In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22861-22872, 2024.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 11-25005-25015 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11315-11322 11355, 2022.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models secretly perform gradient descent meta-optimizers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Associa- fion for Computational Linguistics: ACL 2023, pp. 4005-4019, Toronto, Canada, July 2023. fin.:02...nning--a..2444444 URL
Association for Computational Linguistics. doi: 10. 10.18653/112023.111 https::/acclantee logy Ftng:/22022..nndnng--ee... .247
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 1107-1128, 2024.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 11:19-1366 2015.
Zhongbin Fang, Xiangtai Li, Xia Li, Joachim M Buhmann, Chen Change Loy, and Mengyuan Liu. Explore in-context learning for 3d point cloud understanding. Advances in Neural Information Processing Systems, 36, 2024.
Jun Gao, Ziqiang Cao, and Wenjie Li. Unifying demonstration selection and compression for in-context learning, arXiv preprint arXiv:2405.17062 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors, arXiv preprint arXiv:2310. 15916, 2023.
Alberto Hojel, Yutong Bai, Trevor Darrell, Amir Globerson, and Amir Bar. Finding visual task vectors. In European Conference on Computer Vision, pp. 257-273, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09668 2021.
Jincen Jiang, Qianyu Zhou, Yuhang Li, Xuequan Lu, Meili Wang, Lizhuang Ma, Jian Chang, and Jian Jun Zhang. Dg-pic: Domain generalized point-in-contex learning for point cloud understanding. In European Conference on Computer Vision, pp. 455-474. Springer, 2024.
Yuxin Jiang, Yuchao Gu, Yiren Song, Ivor Tsang, and Mike Zheng Shou. Personalized vision via visual in-context learning. arXiv preprint arXiv:2509.2511 251 2025.
Miaoge Li, Jingeai Guo, Richard Yi Da Xu, Dongsheng Wang, Xiaofeng Cao, Zhijie Rao, and Song Guo. Tsca: On the semantic consistency alignment via conditional transport for compositional zero-shot learning. arXiv preprint arXiv:2408.0877 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
11

---

Published as a conference paper at ICLR 2026
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023a.
Xinyang Liu, Dongsheng Wang, Bowei Fang, Miaoge Li, Zhibin Duan, Yishi Xu, Bo Chen, and Mingyuan Zhou. Patch-prompt aligned bayesian prompt tuning for vision-language models. arXiv preprint AXXiv:2333.0100 2023b.
Trevine Oorloff, Vishwanath Sindagi, Wele Gedara Chaminda Bandara, Ali Shafahi, Amin Ghiasi, Charan Prakash, and Reza Ardekani. Stable diffusion models are secretly good at visual in-context learning. arXiv preprint arXiv:2508. 09949, 2025.
Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In European conference on computer vision, pp. 604-6011111 621. Springer, 2022.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.0771 00779,, 2020.
Congpei Qiu, Yanhao Wu, Wei Ke, Xiuxiu Bai, and Tong Zhang. Refining clip's 's spatial awareness: A visual-centric perspective. arXiv preprint arXiv:2504.02328. 2025.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 15:21112255 2015.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.034: 2017.
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. knn-prompt: Nearest neighbor zero-shot inference. arXiv preprint arXiv:2205.1379/ 2022.
Seongjin Shin, Sang-Wou Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Woo Ha, et al. On the effect of pretraining corpora on in-context learning by a large-scale language model. arXiv preprint arXiv:2204. 13509, 2022.
Yanpeng Sun, Qiang Chen, Jian Wang, Jingdong Wang, and Zechao Li. Exploring effective factors for improving visual in-context learning. IEEE Transactions on Image Processing, 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 2rXiv:2312.1 1805, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXi::2302. 2023.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp. 35151-35174. PMLR, 2023.
Jinpeng Wang, Tianci I Luo, Yaohua Zha, Yan Feng, Ruisheng Luo, Bin Chen, Tao Dai, Long Chen, Yaowei Wang, and Shu-Tao Xia. Embracing collaboration over competition: Condensing multiple prompts for visual in-context learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25 25156-25165. 2025.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6830-6839, 2023a.
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.032888 032844 2023b.
12

---

Published as a conference paper at ICLR 2026
Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang. Mingyuan Zhou, et al. In-context learning unlocked for diffusion models. Advances in Neural Information Processing Systems, 36:8542- 8262. 2023c.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raftel, Barret Zoph, Sebastian Borgeaud. Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently, arXiv preprint arXiv:2303 03846, 2023.
Yongxian Wei, Yilin Zhao, Li Shen, Xinrui Chen, Runxi Cheng. Sinan Du, Hao Yu, Xiaohan Wang. Gang Liu, Jiahong Yan, et al. Learning pose problems: Reasoning-driven and solver-adaptive data synthesis for large reasoning models. arXiv preprint arXiv:2511. 1.0907, 2025.
Chengming Xu, Chen Liu, Yikai Wang, and Yanwei Fu. Towards global optimal visual in-context learning prompt selection. arXiv preprint vrXiv:2405.15277 2024.
Xu Yang, Yongliang Wu, Mingzhuo Yang. Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. Advances in Neural Information Processing Systems, 36: 40924-40943, 2023.
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang- Woo Lee, Sang-goo Lee, and Taeuk Kim. Ground-truth labels matter: deeper look into input-label demonstrations. arXiv preprint trXiv:2205.12685, 2022.
Hao Yu, Jiabo Zhan, Zile Wang, Jinglin Wang. Huaisong Zhang, Hongyu Li, Xinrui Chen, Yongxian Wei, and Chun Yuan. Omnialpha: A weaeeneee-----ueeeee framework for unified multi-task rgha generation. arXiv preprint arXiv: 25/1.20211, 2025.
Hao Yu, Shuning Jia, Guanghao Li, Wenhao Jiang, and Chun Yuan. Enhancing geometric perception in vlms via translator-guide -guiddd reinforcement learning, 2026. URL/attpa://axxivoorr
2602.22708 22103
Dongxu Yue, Qin Guo, Munan Ning, Jiaxi Cui, Yuesheng Zhu, and Li Yuan. Chatface: Chat-guided real face editing via diffusion latent space manipulation. arXiv preprint 233. 2023.
Dongxu Yue, Maomao Li, Yunfei Liu, Qin Guo, Ailing Zeng, Tianyu Yang, and Yu Li. Addme: Zero-shot group-photo synthesis by inserting people into scenes. In European Conference on Computer Vision, pp. 222-239. Springer, 2024.
Jiahao Zhang, Bowen Wang, Liangzhi Li, Yuta Nakashima, and Hajime Nagahara. Instruct more! random prompting for visual in-context learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2597-2606, 2024.
Jiahao Zhang, Bowen Wang. Hong Liu, Yuta Nakashima, and Hajime Nagahara. Panicl: Mitigating over-reliance on single prompt in visual in-context learning. arXiv preprint arXiv:2509.219926 2025.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36, 2023.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv preprint ar0iv:2305. 12740, 2023.
Xingyu Zhu, Shuo Wang, Beier Zhu, Miaoge Li, Yunfan Li, Junfeng Fang. Zhicai Wang, Dongsheng Wang, and Hanwang Zhang. Dynamic multimodal prototype learning vasion-language models. In P Proceedings of the IEEF/CVF international conference computer vision, pp. 2501-2511, 2025.
13

---

Published as a conference paper at ICLR 2026
A SCOPE OF LLM USAGE
To remain compliant with responsible LLM usage protocols, we limited the scope of LLMs to improv- ing readability and grammar. Every scientific contribution, including the conceptual development, experimental design, and analytical validation, was independently carried out and confirmed by the authors, and we retain complete responsibility.
B FUTURE WORKS AND LIMITATIONS
B. WHITE-BOX DEPENDENCY
Like CONDENSER, PromptHub requires access to the backbone's parameters and gradients to train the fusion module, even though the backbone itself is frozen. This design has been instrumental in ensuring the framework's success and robustness in its current applications. But this makes scaling to very large models or soosed- models challenging, as full gradients may be inaccessible too costly. While this enables superior performance, extending prompt fusion to black-box or gradient-free settings a key direction for future work.
B.2 EXTENDING APPLICABILITY TO LINGUISTIC AND MULTI-MODAL DOMAINS
PromptHub is designed for VICL tasks with constrained inputs, utilizing positional correspondences between query and label image patches for locality prompt fusion. Building on its success in the visual domain, future work will expand its scope to multi-modal scenarios by exploring generalized mechanisms that effectively align visual and linguistic modalities, enabling broader applicability and integration.
C PRELIMINARY: MAE-VQGAN
As described Figure g MAE-VQGAN (Bar et al., 2022), comprising the MAE (He et al. 2022) and VQGAN components, serves as backbone for VICL through in-painting ap- proach. Given an example and query for the current task, MAB-VQGAN is treated satile model capable of solving several image- to-image tasks.
(o/MAE VOGAN's Training
Cross Entropy
VQGAN Encoder
Encoder
Lows
Rondom
CVF Dataset's
Mask
Origin Image
(bjMAEE-VGAN's Inference
Get
fied Tokend
During the pre-training phase, the model is trained on a dataset CVF, where each image is constructed from multiple sub-images, pro- ceeding the masked reconstruction task. This process fine-tune the MAE encoder to align the distances with its the VQGAN's codebook space. In the inference phase, in-context sample is fed into the MAE encoder, and corresponding content from the VQGAN's codebook space is obtained, which is then passed to the VQGAN decoder for generating the output.
Encoder
VOGAN
Mash the Part
Generated
Lobel
Figure 8: Introduction to MAE-VQGAN (Bar et al. 2022); (a) the pre-training stage, MAE (He et al. (002) is trained to enhance its inference capability through a masked reconstruction task on CVF dataset. (b) In the inference stage, the prompt pair is placed above, with the query positioned be- low, and both are fed into the model for generative processing.
We utilizes the pre-trained parameters of MAE- VQGAN, freezing its parameters throughout the entire process.
14

---

Published as a conference paper at ICLR 2026
D INFERENCE TIME AND GPU OVERHEAD
As shown in Table 7 we compare the inference time and GPU usage of PromptHub with other baselines. The time for retrieving prompt pair included in the inference time. All meth- ods that require only prompt pair retrieval are categorized under the MAE-VQGAN class. It can be observed that the time overhead of our approach increases only modestly compared to other methods, with GPU usage growing ap- proximately 30MB per prompt pair. Therefore, PromptHub is resource-efficient. This further confirms the lightweight nature the plug-in PromptHub based prompt fusion, which in- curs only minimal additional computational and GPU overhead. The study underscores the practical
Table 7: Comparison of the inference time and
GPU overhead between PromptHub and baselines.
Inference Time (ms/query)
GPU Cost
Method
(MB/query)
MAE-VQGAN
51.26
416.14
InMeMo
54.28
497.50
Prompt-Sefe Seaot-SelF--111 N=16
984.62
441.75
NONDENSERN=1
59.17
565.42
CONDENSER N=16
66.61
1021.66
PromptHuby=1
63.14
569.88
PrompHHuby=1e
70.40
1032.50
feasibility of deploying this approach in real-world scenarios, offering effective and resource- efficient solution.
E ANALYSIS OF HYPERPARAMETER
N=1
N=16
E. ANALYSIS OF HYPERPARAMETER o
47.2
0
The hyperparameter influences the neighbor- hood range selected by PromptHub. When
44.2
45.2
43.2
0
0.3
0.5
0.65
0.8
1.0
1.5
2.5
- O, the selected neighborhood consists solely of the content of the current (h, w) to- ken. As
(a) Mean Performance Segmentation
oc, the selected neighborhood encompasses global information, equivalent to the standard cross-attention. As shown in Fig-
N=1
N=16
46
3 44.4
45.4
0
9 extremely large or small values of a result
43.4
in either insufficient emphasis on local informa- tion neglect of global information. Moreover, the optimal a value varies across tasks. For high- level and low-level tasks, 0.5 and 2.5 are both reasonable choices, respectively.
0.3
0.5
0.65
0.8
1.0
1.5
(b) Performance on Detection
N=1
N=16
0.51
0.53
F EXPERIMENTAL ANALYSIS OF QUERY-CONDITIONAL SIGMA
0.55
1.5
2.5
3.5
5.0
10.0
(c) Performance on Image Coloring
We design a straightforward query-conditioned sigma mechanism investigate the impact of adaptive for the same task. Specifically, we average the embedding dimension of the query [batchsize, patch- patch-number, embeddim], apply linear layer, and use a sigmoid activation to constrain the sigma value within (0,1). report its performance on segmentation and detection tasks.
Figure 9: Evaluation of PromptHub's 's performance
on three tasks across varying values of o.
Table Comparison of results between query-conditioned sigma and hyperparameter sigma.
Method
Fold-0
Fold-1
Fold-2
Fold-3
Mean
Det
PromptHuhx adaptive sigma)
43.79
51.93
44.56
43.18
45.86
44.25
sompptHaox--uuqqutya-dddd sigma)
46.44
52.97
45.66
46.89
47,99
45.41
PromptHubx--chhyyyprrrmnee sigma)
43.26
50.75
43.83
42.82
45.17
44.51
PrompdHub x wtwp:/uubbbboddlyyypppaaeeee sigma)
45.93
53.12
45.44
46.74
47.81
45.59
As shown in Table 8 employing simple learnable a within the same task yields limited improve- ments. This suggests that more sophisticated spatially varying priors are required, which we leave for future exploration.
15

---

Published as a conference paper at ICLR 2026
Query
Map for Prompt
Map for Prompt 2
Query
Map Prompt
Map for Prompt 2
Query
Map
Map Promet
Query
Map for Prompt
Map for Prompt 2
0000
Figure 10: Visualization of attention map for N 2.
G EXPLORING MULTI-OBJECTIVE SEGMENTATION
Table
Results of Multi-Objective Segmentation Experiments.
Method
Fold-0
Fold-1
Fold-2
Fold-3
Mean
SupPR
26.85
32.73
33.48
28.40
30.37
InMoMo
28.
38.31
37.94
33.08
34,37
PromptHubx-ia
38.56
46.54
45.34
39.23
42.41
We further report the numerical results on multi-object segmentation, using subset filtered by anno- tations. As shown Table on the complex task of multi-objective segmentation, our PromptHub model achieves an average mloU that surpasses the strongest competitor, InMeMo, by approximately 23.4%. This demonstrates that our approach maintains strong transferability in challenging tasks and exhibits robust generalization capability.
H MORE VISUALIZATION
H.1 VISUALIZATION OF VICL ANSWER VIA PROMPTHUB
As illustrated in Figure II PromptHub consistently outperforms prior baselines across all three tasks. In particular, the segmentation and colorization results demonstrate that the predictions generated by PromptHub exhibit smoother textures, which further substantiates the advantages of the locality- aware paradigm. Moreover, the ability of PromptHub strengthen multi-prompt VICL highlights its potential to drive more comprehensive progress in this domain.
H.2 VISUALIZATION OF ATTENTION MAP
we visualize the attention map for prompt fusion with = 2, demonstrating that PubmptHub effectively focuses regions corresponding to the query image. The attention score for the current patch is computed as the normalized result of its attention score from all query patches.
As shown in Figure T0
16

---

Published as a conference paper at ICLR 2026
Query
MSE: 0.88
MSE: MSE: 1.20
MSE: 0.04
MSE: 0.86
MSE: 0,98 MSE: 1.47
MSE: 0.93
MSE: 1.13
InMeMo
MSE: 0.77
MSE: 1.04
MSE: 0.89
MSE: 0.69
MSE: 0.68
MSE: 0.54
MSE: 1.26
MSE: 0.82
MSE: 0.93
Condenser N=16
MSE:
0.72
PromptHub N-16
Ground Truth
Query
Tol: 23.63 loU: 40.21 tol: 29.99 1oU: 29.35
1oU: 28.33 1o 50.64 loU: 21.73 IoU: 35.09 loU: 32.00
InMeMo
lol: 42.63 lot: $1, 0..06 tot: 38.19 19-19.10 lol:: 42.93 Jol: 60. tol:: 60.56 Iok:: 69,20 Inl:: 48,99 tntl: $2.94
Condenser N=16
1ol: 70.38 lou: 72-68
toU: 65.23 JoUs 65.24 JoU: 75.69 toV: 85.25 JoU: 82.05 Joit 76,35 loti 63.28
PromptHub N-16
Ground Truth
Query
ToU: 49.04 loU: 10.90 loU: 11.90
IoU: 1.77
IoU: 2.04 toV: 43.64
loU: 51.67 loU: 25.83 tot: 20.38
InMeMo
loU: loU: JoU:
Condenser N=16
tou:
lol:
PromptHub N-16
Ground Truth
Figure 11: Comparative visualization of our method against the existing state-of of-the-art arrt method for Foreground Segmentation and Single-Object Detection and Colorization tasks.
17


Reviews:
[{"review_id": "j8CLlJDdwj", "reviewer": "Reviewer_yUCw", "summary": "This paper focuses on enhancing the multi-prompting capability of MAE-VQGAN through the method of prompt fusion. The authors identify two main limitations of the recent prompt-fusion method CONDENSER: it fails to fully utilize local visual cues from multiple prompts, and the discrepancies that arise from prompt fusion can cause the model to revert to its standard inference method. To address these issues, the authors introduce PromptHub, which spatially aggregates relevant information from multiple prompt pairs using cross-attention. Additionally, they implement learning objectives and data augmentation for prompt pairs to help mitigate the discrepancies. Experiments conducted on different benchmarks demonstrate the effectiveness of the proposed method.", "questions": "Please address the concerns mentioned in the weaknesses section.\n\nIn Section 3.4 (ii), random pairs are sampled as prompt pairs to improve the robustness of PromptHub. I am curious if the authors have assessed the model's performance when these prompt pairs are combined with noisy pairs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The concept of utilizing local-aware aggregation for prompt fusion and the design of learning objectives is illustrative and aligned with the original motivation.\n\n- The performance improvement is significant compared to competitors.\n\n- The writing is generally well-structured and easy to understand.", "weaknesses": "- **Interpretability**: As shown in Figure 6, the fused prompts of Condenser align closely with the target regions of the queries. In contrast, the generated object contours in PromptHub are misaligned with the queries. To improve clarity, it would be beneficial to include additional visual results, such as attention maps of the prompt regions similar to those in Figure 13.\n\n- The cross-attention operation is constrained by the coordinate position. What would happen if the prompt pairs have distinct positions compared to the query? I wonder if this situation would harm the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q6HGUeO8aU", "reviewer": "Reviewer_92Kf", "summary": "Focues on the visual in-context learning, this paper proposes PromptHub, a novel learning-based framework to generate the optimal fused examples for each query from N examples. To do this, PromptHub first designs a locality-enhanced cross-attention strategy between the query image and examples to generate the unified prompt, and then develop three alignment loss to train the fusing process. Extensitve results on three vision tasks (segmentation, detection, and coorization) show the effectiveness of the proposed PromptHub.", "questions": "Please see the Weaknesses section above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The task of generating optimal prompt in VICL is a hot topic in the community. And the results of PormptHub show the improvements of this method.\n\n2) Three new learning objectives are developed to train the fusing process.", "weaknesses": "1), One of the main concerns lies in the novelty of the work. Given that the previous CONDENSER model also generates fusing prompts from the query image and N pairs, I find that the locality-aware attention and training losses developed in PromptHub may limit the novelty of this paper. I hope the authors can clarify the core differences between the previous work and PromptHub in terms of the main motivation, fusing strategy, and training process.\n\n2), Table 3 presents ablation results demonstrating the effectiveness of the designed modules, which helps readers understand their contributions. However, I notice that when using the global fusing strategy (a traditional approach), the performance of PromptHub is lower than that of CONDENSER. Does this suggest that the three newly introduced losses do not effectively improve alignment? The authors should provide a deeper discussion of this issue.\n\n3), What is the motivation behind the three proposed losses, particularly L_s and L_u ? These losses might negatively affect the fusing process if their hyperparameters are not properly tuned.\n\n4), Table 2 reports cross-dataset results of PromptHub, showing the strong transferability of the proposed model. I am also interested in its cross-task performance—specifically, when the model is trained on segmentation but tested on detection. This setting would be highly valuable to the VICL community, as it evaluates the model’s ability to perform in-context learning on unseen tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "APY3rxWm0d", "reviewer": "Reviewer_3dRG", "summary": "In this paper, the authors propose PromptHub, a locality-aware multi-prompt fusion framework for Visual In-Context Learning (VICL). By integrating three cooperative losses (semantic alignment, utilization, and prediction), PromptHub addresses the shortcomings of patch-wise fusion methods like CONDENSER.", "questions": "Could the authors provide some failure cases or qualitative examples where PromptHub does not perform well, and analyze the possible reasons behind these failures?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall idea is well-motivated, as the paper clearly identifies the limitation of patch-wise fusion in existing multi-prompt VICL methods and introduces a locality-aware fusion mechanism that intuitively enhances spatial coherence and prompt utilization.\n\n2. Extensive experiments showing consistent gains across segmentation, detection, and colorization.", "weaknesses": "1. The claim that PromptHub ‘establishes an interpretable paradigm’ is somewhat overstated. Moreover, the analysis of interpretability remains superficial, lacking quantitative evidence or deeper reasoning about what the fused representations capture.\n\n2. The visualization results are not entirely convincing.  For instance, in some examples (e.g., prompt 1), background regions such as the sky or ground areas from unrelated prompts (e.g., prompt 2) are also highlighted, suggesting that the fusion may still mix irrelevant spatial cues rather than providing truly interpretable alignment.\n\n3. The replacement probabilities $p_q$，$p_r$ are not justified, and the approach lacks ablations on different augmentation ratios. It’s unclear whether improvements come from the locality-aware fusion or the added stochastic augmentation.\n\n4. The paper does not explicitly discuss the limitations of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)