VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion
Authors: Jaekyun Park
Hye Won Chung
Venue: NeurIPS 2025 poster ()
Keywords: Parameter-efficient fine-tuning
Prompt tuning
Transfer learning
Primary area: deep_learning
TL;DR: VIPAMIN is a lightweight visual prompt initialization method that enhances self-supervised model adaptation by aligning prompts with semantic regions and expanding representational diversity.

Abstract: In the era of large-scale foundation models, fully fine-tuning pretrained networks for each downstream task is often prohibitively resource-intensive. Prompt tuning offers a lightweight alternative by introducing tunable prompts while keeping the backbone frozen. However, existing visual prompt tuning methods often fail to specialize the prompts or enrich the representation space--especially when applied to self-supervised backbones. We show that these limitations become especially pronounced in challenging tasks and data-scarce settings, where effective adaptation is most critical. In this work, we introduce VIPAMIN, a visual prompt initialization strategy that enhances adaptation of self-supervised models by (1) aligning prompts with semantically informative regions in the embedding space, and (2) injecting novel representational directions beyond the pretrained subspace. Despite its simplicity--requiring only a single forward pass and lightweight operations--VIPAMIN consistently improves performance across diverse tasks and dataset sizes, setting a new state of the art in visual prompt tuning.

VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion
Jaekyun Park
Hye Won Chung"
School of Electrical Engineering KAIST
School of Electrical Engineering KAIST
jaekyun baakyun.parkkkasst.ac.. aa.
hwchungkkaist.ac..
Abstract
In the era large-scall scale foundation models, fully fine-tuning pretrained networks for each downstream task is often prohibitively intensse Prompt tuning offers a lightweight alternative by introducing tunable prompts while keeping the backbone frozen. However, existing visual prompt tuning methods often fail to spe- cialize the prompts or enrich the representation space-especially when applied self-supervised backbones. We show that these limitations become especially pro- nounced in challenging tasks and data-scarce settings, where effective adaptation is most critical. In this work, introduce VIPAMIN, visual prompt initializa- tion strategy that enhances adaptation of self-supervised models by (1) aligning prompts with semantically informative regions in the embedding space, and (2) injecting novel representational directions beyond the pretrained subspace. Despite its pepplicity-rqquirnng only single forward pass and lightweight operations- VIPAMIN consistently improves performance across diverse tasks and dataset sizes, setting state of the in visual prompt tuning. Our code is available https://ggtthe com/ / immj catuuu//viaiinnnn
Introduction
Large-scale Vision Transformers (ViTs) have become central to modern computer vision. Tradition- ally, adapting pretrained model to new downstream task involves full fine-tuning, updating all model parameters and a task-specific head. However, as foundation models continue to scale, full fine-tuning incurs substantial computational and memory overhead.
Prompt tuning offers a more efficient alternative by introducing a small number of trainable tokens (prompts) while keeping the backbone frozen. Initially successful in NLP, prompt tuning has shown strong performance in vision through Visual Prompt Tuning (VPT) [27], which rivals full fine-tuning
on many benchmarks using only few dozen tokens (27 36 37 58
VPT has also proven effective and continual learning 26 67
more challenging settings such as test-time adaptation
While recent theoretical work shows that prompt tokens can selectively modulate attention to empha- size relevant inputs [48). and may serve as retrieval mechanisms for pretrained knowledge [5] 65
these findings do not fully explain VPT's limitations in practice-especially with self-supervised backbones. Empirically, we identify two key failure modes such settings: (1) prompts exhibit near-uniform attention across tokens, lacking specialization, and (2) prompt outputs collapse into the pretrained self-attention subspace, failing to introduce novel task-relevant features. These limitations severely hinder VPT's 's adaptability, especially under distribution shifts and in low-data regimes where representational diversity and efficient adaptation are essential.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Motivated by these findings, we introduce VIPAMIN: Visual Initialization of Prompts via Attention- guided Matching and Injection of Novelty. VIPAMIN a lightweight initialization scheme requiring only single forward pass through the model and two inexpensive matrix operations. It comprises two components designed to address the key failure modes observed in vanilla VPT. The matching module aligns each prompt with semantically coherent input regions, mitigating the issue of uniform attention. The orthogonalizing module projects the prompt away from the row space of the frozen self-attention output, introducing novel representational directions and alleviating subspace collapse. VIPAMIN adds no additional learnable parameters, incurs minimal computational overhead, and integrates easily into existing VPT pipelines with one-time initialization step.
We rigorously benchmark VIPAMIN on 19 vision tasks covering natural, specialized, and structured recognition-using two leading self- supervvee backbones (MoCo-v3 7 and MAE (2T). Additionally, we assess effectiveness under data-scarce conditions on five few-shot benchmarks. Concretely, for MoCo-v3 pretrained model, VIPAMIN achieves 3.7% gain over full fine-tuning on tasks that deviate significantly from the pretraining distribution (i.e., Structured), and 4.6% gain on tasks more closely aligned with the pretraining domain (i.e., Natural)-establissing the new state-of-the-an performance for visual prompt tuning without introducing additional parameters, computational latency, memory overhead. In few-shot settings, VIPAMIN consistently outperforms the baseline across all datasets. Our contributions are summarized as follows:
We identify underexplored limitations of self-supervised VPT-uniform attention and subspace collapse- arising from the internal dynamics of the transformer architecture.
We introduce VIPAMIN, lightweight, non-intrusive initialization technique that addresses these issues via attention- -uudded matching and orthogonal subspace injection.
We demonstrate that VIPAMIN is readily deployable, computationally efficient, and consis- tently enhances performance across wide range of datasets,
Related Work VPT has shown suboptimal performance when applied to self-supervised models. GatedPT improves visual prompting by facilitating inter-block interactions of ViT, demonstrating effectiveness in such self-super settings [70]. The work on SPT, most closely related to ours, shows that prompt representations tend collapse onto the embedding tokens during training and actively leverages this property by initializing prompts with downstream token prototypes to improve VPT's convergence [66). iVPT enhances prompt-embedding interplay via attentive reinforcement module that steers prompts toward salient tokens (76). while VFPT reduces the domain gap by modulating a subset of prompts in the frequency domain using a 2D Fast Fourier Transform (72). Most recently, DA-VPT selects prompt membership based on the CLS token of same-class images, optimizing the assignments via metric learning 53
Unlike prior methods that add learnable gates (GatedPT), reinforcement blocks (iVPT), metric learning (DA-VPT), or Fourier transforms (VFPT), VIPAMIN modifies only the initial prompt weights-incurring zero overhead. Whereas SPT also incurs no additional cost in training, its accuracy hinges on costly offline clustering step that is hard to amortize, while VIPAMIN exceeds the performance of SPT with only a couple of lightweight matrix operations.
2 Preliminaries: Roles of Prompts in Visual Prompt Tuning
We use the following notations throughout the paper. A bold lowercase character (e.g.. x) denotes vector, while a bold uppercase character (e.g., P) denotes a matrix. The i-th row and the (ih entry of the matrix P are represented by (P); and (P)s respectively. Let (-,-) denote column- concatenation, and [-: denote row-wise concatenation. Let [n] := {ili E Ni
VIT and VPT VIT 13 processes input image by dividing it into fixed number of non- overlapping patches, each corresponding to a token the transformer. For an image x E g3xhx and patch size (N', w'), the number of resulting tokens is N == A x w +1, including the additional *class token' (CL.) used in image classification. These tokens are passed through PatchEmbed module and added by learnable positional embeddings. We denote the resulting embedding matrix for the Ne input tokens of sample as E, (xI....,XM) XN.) N.d.
"

---

VPT 12 introduces a set of learnable prompts Po = (pI..... PN..T E RN prepended to the input image tokens Eg, enabling lightweight task adaptation while keeping the original ViT frozen. The combined input to the transformer is denoted by Zo [Po: Eø] E [B(N,+,,))xd VPT has two common variants: VPT-Shallow, in which the prompt Po is prepended before the first transformer block, and VPT-Deep, which injects distinct, learnable prompts before each transformer block. In this section, we focus on VPT-Shallow and analyze the propagation of the initial prompt Po through the ViT architecture. Assuming the ViT consists of L blocks, let B, denote the operation of the i-th block. The evolution of the prompt and input embeddings through each block is given by:
P X.J = (P,,,:;; X.-))) E R (N),+N for E (L),
(I)
where X,, = Egg Each block consists of a Multi-head Self-Attention (MSA) module followed by Forward Network (FFN), with Layer Normalization (LN) and residual connections. For simplicity, we focus on a single-head Self-Attention (SA) applied to pempp--repended input Zo:
Zo W WoooZoWW)) Vd
SA)) softmax
Zo Wv (N,,+N,)xd
(2)
where Wo. WK. K. Wv E Rdxd are the query, key, and value projection matrices, respectively. The softmax applied row-wise to normalize the attention weights.
To better understand the role of Po. we decompose the matrices involved. For Zo = [Po: Xo]. let
Q Zo WQ [Qp, Qx.1: K Zo K
[Kp, Kx,l: V Zo [Vp.: Vxal
The attention matrix S and the resulting SA output can then be written as:
S softmax Vd
[QKT
softmax (/a
1 OPp K
Po
Qx 2x.
QP. Qr.Kk.]
Sp Sx
SpoXo Sx
; (3)
Qx. Kp.
SpoPo
SpoXo
Po v X Wy
= (P. (Po Wv) Sx.x, (Xn Wv) E (N
(P. v) + SP,x, (X. Wv)
SA(Za) =
Sx.Pa
Sx.X.
Note that in the absence of prompts Po. the self-attention output for the original input tokens Xo reduces to SA(X,) Sx, (X, E Nex, where Sx, denotes the attention weights computed solely among the input tokens. Comparing this to the prompt-augmented case reveals two distinct roles played by prompts in modifying the output of the self-attention module: (1) The prompts Po attend to the input tokens Xo. filtering, selecting, and aggregating relevant information through Sp.x, (Xo W.), which then passed to the next block via the prompt outputs. (2) In addition, the prompts inject new information into the input token representations by contributing a prompt-induced bias term Sx,P, (Po Wv). thereby modifying the self-attention output for the original tokens.
Role 1: Selecting Tokens for Prompt Propagation The first Np rows of SA(Z,) correspond to the updated prompt representations after a ViT block. As described in ( these outputs are propagated through the transformer in VPT-Shallow. The effectiveness of VPT hinges on the prompts' ability to selectively aggregate and transmit informative content from the input tokens. Each row of Sp,x, (X, Wv) E NR xd forms a weighted combination of input embeddings, where the weights are given by the attention scores in Sp. Xo. Since only a subset of input tokens are typically task- relevant, selective attention is essential for filtering noise and propagating meaningful information.
Role 2: Injecting Semantic Bias into Attention Outputs Prompts also modulate the output of input tokens by introducing prompt-induced bias. The updated self-attention output for input tokens (the last N₂ rows of SA(Zo)) is given by:
Px (Po Wv) + Sx,x, Wv) = Sx,P, (Po Wv) + diag(1N. - Sx.P. I )s,(X))))). (4)
where SA(Xo) is the self-attention output without prompts. This shows that prompts inject linear bias into the attention output, potentially shifting representations toward task-relevant semantics 25 51 In the next section, we empirically evaluate whether VPT effectively fulfills these two selas-semantic token selection and semantic bias -njectio in challenging downstream tasks and settings.
3

---

Disbddar
34.8%
-9.5%
-0.72%
Dissimilar Serted VTAB Datasets [UP/FT!
Smi
FJll
Few-Shot
Few-Shot
(a)
(b)
(c)
Figure (a) Nineteen VTAB-Ik tasks ordered by a proxy for task similarity to the pretraining dataset, measured by the relative accuracy ratio of linear probing compared to full fine-tuning (LP ratio). Representative images from four datasets- dSpietts-Ss BmalINORR/azii, Caltech101, and Sun397-and are shown to the right. (b) Relative accuracy ratio of VPT compared to full fine-tuning (VPT ratio) across 19 VTAB datasets, sorted by the LP ratio. (c) Accuracy comparison of VPT between training with the full dataset and few-shot learning (8 images per class), evaluated on the CUB-200-201 dataset using supervised and self-supervised (MoCo-v3) pretrained weights.
3
Motivation
To evaluate whether prompt tuning effectively leverages its two core mechanisms-semare token selection and semantic bias injection-we focus two settings where these capabilities especially critical: (i) distribution-shiie
tasks, where the downstream task diverges significantly from the pre- training domain, making both semantic alignment and the injection of novel task-specific information essential; and (ii) few-shot scenarios, where limited labeled examples demand that prompts focus attention informative regions and introduce meaningful biases to enable rapid adaptation.
For distribution-shifted tasks, we assess VPT's performance across the 19 classification tasks in the VTAB-Ik benchmark (73), which includes Natural (photographic), Specialized (e.g.. medical or satellite), and Structured (geometric reasoning) categories. Following prior work [33 42 we order the tasks by similarity to the pretraining distribution (ImageNet [10). using the performance ratio between linear probing and full fine-tuning (LP/FT) as proxy: larger ratios indicate stronger alignment, while smaller ratios suggest greater divergence. Figure I(a) shows the resulting task ordering, from most dissimilar (dSprites/loc) to most similar (Sun397).
Figure 1(b) reports the relative performance of VPT compared to full fine-tuning, measured as (AccyPT Accrun) Acccan)/ACC..ll AcCuul across the ordered tasks. The results show that VPT performs strongly on the top 30% most similar tasks (averaging +7.6%) but suffers sharp performance degradation on bottom 30% most dissimilar tasks (averaging 34.6%), where greater representational shifts are required. This highlights VPT's limited adaptability under distribution shift.
In the few-shot setting, we compare VPT's performance with training samples per class against full-data VPT on the CUB-200 LUB-200-2011 dataset [59], using both supervised [13] and MoCo-v3 [ self-supervised pretrained backbones. full-data counterpart in both cases: the supervised backbone meurs a modest drop of 9.5%, while the supersupe backbone suffers a much larger drop 40.72%. This pronounced decline highlights the challenge of few-shot adaptation, especially for self- sufe-uueeee models.
shown in Figure [[c], few-shot VPT underperforms its
Together, these results reveal that VPT struggles adapt in settings where its core mechanisms- focused attention and representational enrichment-are most essential. In the following section, we quantitatively analyze VPT's behavior with respect semantic token selection and bias injection, and demonstrate that its failures in these challenging scenarios stem from underutilization of these mechanisms. This motivates closer investigation into how better activate and enhance these roles.
Prompts Fail to Specialize To evaluate whether prompts attend to meaningful input tokens, we examine the attention matrix Sp,x, E RRNxNN. from 3 which encodes the attention weights from each prompt to the input tokens. Since identifying truly semantic tokens would require additional supervision (e.g., segmentation labels), instead use the concentration of attention as proxy for

---

Premot Attention Entropy
5.25
0.98
Projection Energy
5.15
0.94
Max Entropy
VPT
VPT
0.90
VIPAMIN
4.05
Dissimilar
Similar
40
80
100
VTAB (LP/FT)
Epach
(a)
(b)
Figure 2: Failure modes of VPT. (a) Prompt attention entropy of the fully trained model for each VTAB dataset, sorted by its LP ratio. The topmost blue line represents the maximum prompt attention entropy attainable, In (. (b) Subspace projection energy of Po Wy onto SA(Xo). tracked during training on dSprites/loc (the most dissimilar task). Both figures are the training results of MoCo-v3 pretrained model.
specialization. For each prompt pi, compute the entropy of its normalized attention distribution:
No. H(p.) (Sp.x.)is
In (SPoxo)is
N.
S
SP,x,
S
where (Sp.xo)is x exp P. W W / This quantity is upper bounded by In(N.): lower values indicate more concentrated (i.c., specialized) attention.
Figure 2(a) reports the average prompt attention entropy across all prompts for each VTAB-1k task, computed from 256 randomly selected training images and extracted from the final ViT layer. We observe that VPT consistently yields near-maximal entropy across tasks, regardless of their similarity the pretraining distribution. This indicates that VPT fails to specialize prompt attention and instead distributes focus uniformly over all input tokens. While such broad attention may suffice for tasks closely aligned with the pretraining domain, limits adaptability more dissimilar tasks, where selective attention is needed to suppress irrelevant features and amplify task-specific signals.
To evaluate whether prompts introduce novel repre- sentational directions into the embedding space, we examine the relationship between Po Wy and (X(X,). As indicated (4). Po Wi contributes a prompt-dependent bias to the self- attention output. However, if the row space of P, Wv v is contained within that of SA(X))-ie., span( (Po W T) S span(SA t-then prompts fail to expand the representation space beyond what already captured by the frozen backbone. To quantify this, we compute the projection energy, which measures the fraction of prompt signal contained within the pretrained self-attention subspace. Wv) and (SA(Xo)) we define:
Prompts Fail Inject Novel Semantic Bias
Given A
| ||A 12
ProjectionEnergy (A B
where PB = B(B BB.
(6)
value near indicates that prompts contribute little to new representational directions, while a lower value suggests successful injection of orthogonal information. Figure ((b) plots the projection energy of Po PoWy Wy y onto SA(Xo) during training on dSpritcs/loc-the most dissimilar task in VTAB- lk. VPT converges to projection energy near revealing that the prompt output collapses into the frozen subspace and fails to provide novel signals needed for adaptation.
Together, these findings show that VPT fails to specialize attention or introduce task-specifa semantic bias capabiiiiis that are particularly critical for cirtiibutionsssiiffe and few-shot settings. To overcome these limitations, we introduce VIPAMIN: Visual Initialization of Prompts via Attention- guided Matching and Injection of Novelty. VIPAMIN initializes prompts by aligning them with semantically relevant input embeddings while injecting directions orthogonal to the frozen subspace. As shown Figures 2(a) and 2(b) (red curves), VIPAMIN produces lower attention entropy and projection energy dissimilar tasks, enabling more effective adaptation in challenging scenarios.
5

---

4 Methodology: VIPAMIN
We propose VIPAMIN, novel initialization method for visual prompt tuning that enables prompts to (I) specialize their attention and (2) inject novel representational directions beyond the pretrained subspace. VIPAMIN consists of two complementary modules: Matching module, which addresses the issue of uniform attention by aligning each prompt with semantically coherent input tokens from the downstream task, thereby promoting specialization over meaningful local regions; and the Orthogonalizing module, which prevents representational collapse by projecting prompts away from the pretrained embedding subspace, facilitating the injection of task-specific information.
Prompt Specialization via Semantically Coherent Token Matching The matching module encourages each prompt focus on semantically coherent local regions from its initialization, with controllable locality. A key challenge is identifying such coherent regions without external supervision (e.g.. segmentation labels). To address this, we leverage a pretrained VÝT and extract the embedding matrix Ejacch through the frozen model. After mean pooling across the batch, we obtain E, Ne
Matching Module:
IR
by passing B downstream training images
Next, for each randomly sampled prompt P: E IRd (initialized using Xavier uniform [17] we project both the prompt and the mean- -pooeed input embeddings into the key subspace W K E Rdxd of the first transformer block. We then compute cosine similarity between the projected prompt and each input token, and select the top-k most aligned tokens:
{az) TopKIndicees(cos(p, W Eo (K)Wk).. k)
where TopKIndi k) returns the indices of the top-k values. In ViT-B/16, each token covers only 0.5% of the image, so semantically coherent regions typically span multiple tokens, which tend to cluster in the key space. This makes it likely that the top-k tokens share similar semantics. Additional in-depth discussion is available in Appendix L.3
shows an example where two ran- domly sampled prompts select distinct token groups. Each group focuses on different se- mantically coherent components of the im- age (e.g., display racks, top/bottom shelves of bread), confirming that random prompts select diverse yet meaningful regions.
Figure TigureBshows
The initialization vector for each prompt is obtained by averaging its matched token em- beddings:
k
avg P
(Eo)a.
(8)
Figure 3: Coherence of embeddings in key space. For an image of a bakery, we show matched tokens for two randomly sampled prompts by highlighting the top-k indices (k = 60) in red.
k
j=1
This encourages each prompt to focus on a coherent semantic region. The hyperparam- eter l controls the locality of this focus. With a sufficient number of prompts (Np)). their union ensures broad coverage of salient image regions. This matching-baat based initialization is fully unsupervised and requires no additional annotation.
Injecting Novel Representational Bias While the matching module initializes prompts by directly leveraging existing semantic clusters in the embedding space, this alone may be insufficient for tasks that diverge significantly from the pretraining distribution. In such cases, prompts initialized solely from existing embeddings may cause the self-attention output in (4) to collapse into the subspace of SA (Eo). limiting representational diversity. To mitigate this, introduce an orthogonal component into each prompt that lies outside the row space SA(Eo).
Orthogonalizing Module:
Specificallyy we compute the SVD of SA(Eo) UΣVT, where V spans the row subspace of SA(E.). We then project a random prompt through the value matrix Wy, remove its projection onto the subspace, and reverse-map it back via the pseudoinverse of Wv:
orth p
f (I - VV (p.
(9)

---

Table I: Fine-tuning results for VTAB-Ik benchmarks with pretrained ViT-B/16. The best and second- best results are highlighted in bold and underline, resp. Per-task results available Appendix F
MoCo-v3 pretrained ViT-B/16
MAE pretrained ViT-B/16
Method
Natural Specialized Structured
Mean
Method
Natural Specialized Structured
Mean
Full
71.95
84.72
51.98
66.23
Full
59.31
79.68
53.82
61.28
VPT
67.34
82.26
37.55
57.94
VPT
39,96
69.65
27.50
40.96
GateVPT
74.84
83.38
49.10
65.80
GateVPT
47.61
76.86
36.80
49.22
SPT
74.47
83.93
55.16
68.33
SPT
62.53
80.90
53.46
62.58
VIPAMIN
76.75
84.14
56.68
69.86
VIPAMIN
62.60
79.96
57.47
64.09
Each final prompt is a weighted combination of the matched (semantic) and orthogonal components:
pi
VIPAMIN + (1 - A)P:" + Ap: arth
(10)
where À E (0, 1] controls the strength of orthogonalization. This hybrid strategy enables prompts to both specialize on meaningful regions and expand into new representational directions. Empirically, higher À values are beneficial for tasks that are semantically distant from the pretraining domain.
5
Experiments
5.1 Experiment Setup
Datasets and Models We evaluate our method on two image classification benchmarks: Visual Task Adaptation Benchmark (VTAB-1k) [3] and Fine-Grained Visual Categorization (FGVC). To assess performance under varying degrees of distribution shift from pretraining, we use VTAB- 1k, which comprises 19 visual classification tasks grouped into three categories: Natural; consisting of everyday images captured with standard cameras, Specialized; containing images taken using domain-speciffe equipment, and Structured; involving tasks that require geometric understanding. For few-shot evaluation, we construct k-shot settings by sampling the training set of five FGVC datasets:
[59], NABirds [56]. Stanford Dogs [30], Stanford Cars [15], and Oxford Flowers [46). While these datasets are relatively casier under full supervision compared to VTAB-Ik.
they effectively reveal the limitations of self-supervised VPT in data-scarce regimes, as illustrated in
Figure I(c) Additional benchmark details provided in Appendix D
We use ViT-B/16 as the backbone model unless otherwise specified. Our method evaluated using two widely adopted self-supervised pretrained backbones: Momentum Contrast v3 (MoCo-v3) 7 and Masked Autoencoders (MAE) 21. More experimental details are reported in Appendix D
Baselines We use VPT 27 and SPT 66 as our main baselines. VPT initializes prompts randomly using Xavier uniform initialization 18 SPT performs K K-means clustering on reshaped input tokens E, E BRBN,xt, xd extracted from Eg E R Nx and uses the resulting Np centroids prompt initializations. Due to the high computational cost clustering in SPT (e.g., 27 days on CUB- 200-2011), 1). we instead adopt low-cost variant that randomly samples N,, tokens from E-- which matches SPT's reported accuracy (Table 4(b) in (66). We refer to this variant as SPT/rand. For SPT/rand, we use the full training set to obtain Eg, while VIPAMIN uses only random mini-batch of size 256. batches from the combined training set and validation set.
FGVC few-shot experiment, both SPT/rand and VIPAMIN sample their respective
5.2 Main Result
Table presents the performance of VIPAMIN, baseline visual prompt tuning methods, and full fine-tuning (Full) across the 19 VTAB- 1k tasks, grouped into Natural, Specialized, and Structured categories. Across both MoCo-v3 and MAE pretrained backbones, VIPAMIN achieves the highest average accuracy. Notably, it delivers substantial improvements on Structured tasks, outperforming VPT by +19.13 % (MoCo-v3) and +29.97 % (MAE), demonstrating its effectiveness in tasks requiring spatial or relational reasoning. Importantly, these gains do not come at the expense of performance on simpler tasks: VIPAMIN also achieves the best or comparable results on the Natural category under both backbones, confirming its broad generalization. For MAE, VIPAMIN is the first prompt-based
7

---

Table 2: Few-shot aceuracy (%) for k-shot classification across five FGVC datasets averaged three independent runs; the final column in each block is the mean accuracy of the five datasets.
CUB Binds Flowers
Dugs Cars
Mean
CUB
Binds Flowers
Dogs Cars
Mean
CUB
Birds Flowers
Dogs
Cars
Mean
CUB
Birds Flowers
Dogs
Cars
Mean
11.7
31.7
37.3
SPTiand
17.2
43.9
22.5
51.7
59.8
21.7
51.7
66.6
92.9
VIPAMIN
28.1
126
528
37.5
5.7
253
36,0
2M
71.6
49.4
11.1
36.2
53.7
41J1
85.1
68.3
21.9
824
68.6
55.1
94.1
70.0
438
66.4
method to surpass full fine-tuning across all VTAB- 1k categories. While it performs strongly on MoCo-v3 as well, falls slightly short in the Specialized group.
Table 2 presents k-shot classification results on five fine-grained benchmarks for k E {1, 4, 8). VIPAMIN consistently achieves the best performance across all settings. In terms of mean accuracy, VIPAMIN outperforms VPT by 7.7% at k: = 1, and the margin grows to 24.7% at k 8. While the gains over the SPT/rand baseline are more modest (1-2%), they remain consistent, underscoring VIPAMIN's superior adaptability in data-scarce regimes.
5.3 Scalability Analysis
Extension to VPT-Deep While our primary analysis has focused the non-intrusive VPT- Sheelow-where prompts are prepended before the first transformer block and propagates-we also evaluate our main ideas on VPT-Deep vari- ant. In VPT-Deep, a separate prompt P, is prepended to the token sequence at each block: ([P Xi)) for l [L 1]. Since each prompt operates only within its corresponding block, the selectien dynamics that are crit- ical in VPT-Shallow are less prominent. How- ever, we hypothesize that prompt-subspace col- lapse can still occur independently within each block's local prompt space (see Appendix [.5] for further discussion).
Table 3: Fine-tuning results for VTAB-1k bench- marks with MoCo-v3 pretrained ViT-B/16. Spe- cialized tasks are abbreviated as Spec.
Method
|| Natural Spec Structured Mean
Full
71.95
84.72
51.98
66.23
Lincar Probing 24
67.46
81.08
30.33
54.69
Partial-1
71
72.31
84.58
47.89
64.61
Bias
52
72.89
81.14
53.43
66.43
Adapter
74.19
82.66
47.69
64.82
VPT-Deep 27
70.27
83.04
42.38
61.22
E 20
76.47
87.28
54.91
69.67
SPT-Deep [66
76.20
84.95
58.36
70.53
iVPT 776
76.12
84.51
57.88
70.21
DA-VPT 53
74.24
83.21
55.23
69.13
In VIPAMIN-Deep. we apply the modules each block's input X₁, using a fixed prompt length of 20. We compare VIPAMIN- Deep against established PEFT methods such as Adapter 4 and Bias Tuning [52], as well
VFPT 72
77.47
85.76
58.74
71.33
VIPAMIN-Deep
|| 77.68 84,79
58.80
71.23
more intrusive approaches that require architectural changes, including V2 20 and iVPT 76 (see Appendix [.1 for a detailed breakdown of the overhead). As shown in Table VIPAMIN-Degs outperforms full fine-tuning across all three VTAB-1k categories, demonstrating strong adaptability. also surpasses the best-known prompt initialization baseline, SPT-Deep. While VFPT slightly out- performs VIPAMIN-Des, in mean accuracy, our method achieves competitive performance without architectural modifications, showing its simplicity and generality.
Scaling Model and Prompt Length A key limitation of VPT is its reduced performance with larger backbones, where increased complexity hinders optimization [27]. similar trend appears with longer prompts, where added tokens often fail to improve representation [3I]. As shown in Fig. (a, b), VIPAMIN mitigates both issues, maintaining stable convergence even with ViT-H/14, where VPT fails. Furthermore, VIPAMIN is the only method that consistently benefits from increased prompt length, with the performance gap widening as prompt capacity grows. This scalability advantage stems from its semantic bias injection, which enables more effective use of additional prompts.

---

Accuracy (5%
80
70
k=
Not Normalize formalized Rccuracyy
40-
Accuracy
65
VIPAMIN
R 32
SPT/rand
k=
30
VIPAMIN
40
VPT
SPT/rand
VPT
35
20
A 1.0
Model Parameters (100)
200
Promot Length (b)
150
Dissimtr
Sorted VTAB Datasets ILP Ratio)
Similar
(a)
(c)
Figure 4: (a) MAE-pretrained ViT-B/L/H are evaluated on Oxford Flowers 102 from FGVC. (b) We assess the effect of prompt length in MoCo-v3 on CIFAR-100. (c) We evaluate the performance of VIPAMIN on MAE across VTAB-IK, varying the hyperparameters and X. Datasets are ordered by LP ratio from lowest highest. We select the learning rate that yields the highest validation accuracy and normalize the resulting top-1 accuracy to the [0. range for each dataset (column wise). A rough trend of the best perffrrefonnng hyperparameter combinations is indicated by the red arrow.
5.4 Ablation Study and Further Analysis
Role of k and X VIPAMIN introduces two key hyperparameters: k in 7 which specifies the number of tokens to which each prompt attends, and À in (1O). which controls the degree of orthogonal bias injected into the prompt initialization. Figure 4 (c) clarifies the roles of l and X. For tasks that exhibit significant distributional shift, smaller and larger À yield the best performance. This suggests that such tasks benefit from together enable the model to focus on task-specific features not captured by the pretrained backbone. In contrast, tasks more closely aligned with the pretraining distribution tend to favor larger k and smaller A, indicating that broader attention and greater reliance on pretrained representations are advantageous.
localized attention and stronger orthogonalization, which
Ablation VIPAMIN Modules We conduct an ablation study to evaluate the contributions of the two core components VIPAMIN: the matching module and the orthogonalizing mod- ule. The results are summarized in Table In- corporating the matching module alone already yields notable improvements over the SPT base- line in the Natural group, indicating that seman- tic token selection is particularly beneficial for tasks that are well aligned to the pretraining do- main. In contrast, tasks in the Specialized group often require the model acquire novel semantic representations (e.g., identifying abnormal retinal features). Here, the orthogonalizing module proves essential by projecting the prompt away from the pretrained embedding space, thereby enabling the injection of new, domain-specific information.
Table 4: Ablation study on the two modules in VIPAMIN, using MoCo-v3 pretrained models eval- uated on the VTAB-1k benchmark.
Matching Orth
Natural
Spec
Structured
SPT baseline
74.47
83.93
55.16
76.50
82.85
56.51
76.75
84.14
56.68
Impact of Orthogonalizing Module Task-Aligned Representation Learning While the or- thogonalizing module ensures that the generated prompt lies outside the pretrained subspace, remains an open question whether this nudging induces representations aligned with task-relevant directions that ultimately improve classification performance. To empirically examine this, we analyze the Last-Layer CLS Representations (LLCR)-i..., the output embeddings of the final trans- former block prior to the classification head. As a reference, we treat the LLCR obtained from fully fine-tuned model the oracle, denoted by LLCR,, We then compare this LLCR and I.CRRIPAMN. which are obtained from models trained using only the matching module and both the matching and orthogonalizing modules, respectively. We focus on the Diabetic Retinopathy dataset in VTAB- 1k, given its clinical nature and the demand for novel Nomain-specinfe knowl- edge. Our evaluation two-fold: instance-level analysis and subspace-level analysis. First, at the instance level, we compute the cosine distance between LLCRRa and each of the compared methods for every validation sample. A Wilcoxon signed-rank test [68] on the paired distances does (LLCRa, LLCR doos (LLCRa, LLCRvIPAMI) yiclds p-value < 2 x 10-7 indicating

---

that LLCR LLCRRvPAMIN is closer to the fine-tuned oracle than LLCRno-otth Second, at the subspace level, we assess representational similarity by viewing the span of each LLCR set as a subspace and com- puting the Grassmannian distance 39 to LLCRA- The distance between LLCRR, and LLC.R is 14.56, whereas the distance between LLCRA and MIIMRN reduced 14.03. These findings suggest that the orthogonalizing module not only increases representational diversity but also actively guides the model toward task-aligned that approximates the fully fine-tuned oracle more effectively.
nudging the representations in direction
Grad-CAM Inspection We analyze Grad-CAM [54] Visualizations for two datasets in VTAB- lk: dSprites/location and Sun397, corresponding to the lowest (most dissimilar pretraining) and highest (most similar) LP ratios, respectively. The results shown in Figure 5 In the case of VPT, we ob- serve a uniform attention pattern, with the model broadly attending most image patches, regardless of the dataset. In contrast, SPT/rand produces more localized attention but often fails highlight seman- tically critical regions. For VIPAMIN, the effect of the hyperparameter & is clearly visible when com- paring k 2 and & 128. A higher & leads to broader attention across the image, while a lower l concentrates attention on narrower region. Conse- quently, for tasks that require localized attention (e.g.. dSprites/location) setting a smaller & is advantageous.
Input
VPT
SPT/rand
k-2
k-128
dSpr/loc
Sun397
Figure 5: Red regions indicate their critical role in the model's class prediction. The cases of = 2 and = 128 correspond to VIPAMIN models trained with their respec- tive hyperparameter, k.
6
Conclusion
We introduced VIPAMIN, a parameter-free initialization scheme for visual prompts in self-supervised models that jointly aligns prompts with semantically meaningful regions and injects novel represen- tational directions orthogonal to the frozen embedding space of vision transformers. Unlike prior approaches that rely on auxiliary modules or architectural modifications, VIPAMIN requires only two lightweight matrix operations, introduces no training overhead, and integrates seamlessly into existing pipelines. Extensive experiments across 24 vision tasks, including data-scarce settings, demonstrate consistent performance improvements.
Acknowledgements
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea govermment (MSIT) (No. RS-2024-00408003, RS-2025-00516153 and RS-2024-00444862,
References
[1] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis,
A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia,
Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M.. Shlens. J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O.. Warden. P., Wattenberg, M., Wicke, M..
Yu, Y., Zheng, X.: TensorFlow: Large-scale machine learning on heterogeneous systems (2015),
https: bttpps://www..c https://sww.teesos// htt////www.eesooffoo org/ software available from tensorflow.org
[2] Beattie, C., Leibo, J.Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq. A., Green, S., Valdés, V., Sadik, A., al.: Deepmind lab. arXiv preprint
(2016)
[3] Bolya, D., Fu, C., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.: Token merging: Your vit but faster. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net (2023), Seramm2iddroZZRaa/e
10

---

[4] Cai. H., Gan, C.. Zhu, L., Han, S.: Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems 33, 11285-11297 (2020)
[5] Caron, M., Touvron, H., Misra, I., J'egou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. IEEE International Conference on Computer Vision (2021). https:///oi.ooo org//0. 10. wtW.ICCCV8992.222..0000
[6] Chen, G. Liu, F. Meng, Z., Liang, S.: Revisiting parameter-efficient tuning: Are we really there yet? In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. pp. 2612-2626, Association for Computational Linguistics (2022). https///do..og/11.. wttp:///rgr..1133///////2200 / 10. 18653/ 20/055/vvv/222 v1/2022. emnlp- mainn. 168
https: org
[7] Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9640 9640099649400 (October 2021)
[8] Cheng, G., Han, J., Lu. X.: Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE (2017)
[9] Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: 2014 IEEE Conference Computer Vision and Pattern Recognition, pp. 3606-3613 (2014). https://do..or//100 10. 10000//2...000000220.0000..
[10] Deng, J., Dong, W., Socher, R., Li, LJJ,, Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.
248-255 (2009). https://doi.or// 10. org/10.
[11] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,
volume (long and short papers). pp. 4171-4186 (2019)
[12] Dong, Y., Cordonnier, J.B., Loukas, A.: Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In: International conference on machine learning. pp. 2793 2803. PMLR (2021)
[13] Dosovitskiy, A.. Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghami, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference Learning Representations (2021), htttp:://ppereviiiie htt.:/tt......ooomooo///.aao..
[14] Fei-Fei, L., Fergus, R., Perona, P.: One-shot learning of object categories. IEEE Transactions Pattern Analysis and Machine Intelligence 28(4), 594-66 994-611 (2006). 1090///M../..000..7999999
[15] Gebru, T., Krause, J., Wang, Y., Chen, D., Deng. J., Fei-Fei, L.: Fine-grained car detection for visual census estimation. In: Proceedings of the AAAI Conference Artificial Intelligence. vol. 31 (2017)
[16] Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. Interna- tional Journal of Robotics Research (2013)
[17] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics. pp. 249-256 JMLR Workshop and Conference Proceedings (2010)
[18] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Teh, Y.W., Titterington, M. (eds.) Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 9. pp. 249-256. PMLR, Chia Laguna Resort, Sardinia, Italy (13-15 May 2010), httpps //proceedings. mlr, prwss/ v9/ v gww.otl0aa...oo html html

---

[19] Gu, Y., Han, X., Liu, Z., Huang, M.: Ppt: Pre-traines prompt tuning for few-shot learning, ACL (2022)
[20] Han, C., Wang, Q. Cui, Y., Cao, Z., Wang, W., Qi. S., Liu, D.: EºV An effective and efficient approach for visual prompt tuning. IEEE International Conference Computer Vision (2023). wtt.d/i....rr0...55555///......
[21] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 16000-16009 (June 2022)
[22] He, K., Chen, X., Xie, S., Li, Y. Doll'ar, ar, P., Girshick, R.B.: Masked autoen- coders are scalable vision learners. Computer Vision and Pattern Recognition (2021).
[23] Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2019)
[24] Iofinova, E., Peste, A., Kurtz, M., Alistarh, D.: How well do sparse imagenet models transfer? In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 12226-12276 (2022)
[25] Jain, A., Chaudhuri, S., Reps, T., Jermaine, C.: Prompt tuning strikes back: Customizing foun- dation models with low-rank prompt adaptation. Advances in Neural Information Processing Systems 37, 47297-47316 (2024)
[26] Jayasuriya, D., Tayebati, S., Ettori, D., Krishnan, R., Trivedi, A.R.: Spare: Subspace-aware prompt adaptation for robust continual learning in llms. arXiv preprint arXiv:2502.02909 (2025)
[27] Jia, M., Tang, L., Chen, B.C., Cardie, C.,, Belongie, S.J., Hariharan, B., Lim, S.: Visual prompt tuning. European Conference on Computer Vision (2022). wttp:io.orrgg...////aa/ii.......
[28] Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L.. Lawrence Zitnick, C., Girshick, R.: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2901-2910 (2017)
[29] Kaggle, EyePacs: Kaggle diabetic retinopathy detection (July 2015)
[30] Khosla, A., Jayadevaprakash, N., Yao, B., Li, F.F. Novel dataset for fine-grained image categorization: Stanford dogs. In: Proc. CVPR workshop on fine-grained visual categorization (FGVC). vol. Citeseer (2011)
[31] Kim, Y. Li, Y., Moitra, A., Yin, R., Panda, P.: Do we really need large number of visual prompts? Neural Networks 177, 106390 (2024). 101.0//////EEEEEET2220...... 2024.10639.
10. org//...016111 neunet
[32] Krizhevsky, A., Hinton, G., al.: Learning multiple layers of features from tiny images (2009)
[33] Kumar, A., Raghunathan, A., Jones, R.M., Ma, T., Liang, P.: Fine-tuning distort pre- trained features and underperform out-of-distribution In: International Conference on Learning Representations (2022), https::/openrrevieet wnt.://orrurr.dddeeee......
[34] LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. vol. pp. I1-104 Vol.2 (2004). https://doi.rr/10.1110 109/CVPR.20000 1315150
[35] Lee, D., Song, S., Suh, J., Choi, J., Lee, S., Kim, H.J.: Read-only prompt optimization for vision-language few-shot learning. In: Proceedings of the IEFE/CV International Conference on Computer Vision (ICCV). pp. 1401-1411 (October 2023)
12

---

[36] Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient
prompt tuning. Conference on Empirical Methods in Natural Language Processing (2021).
https///oii.or//1. 18653/v1/2001 http///2ee.emmmmainn......
[37] Li, X.L., Liang. P.: Prefix-tuning: Optimizing continuous prompts for gener- ation. Annual Meeting of the Association for Computational Linguistics (2021). hyps://doi.org/100/ wrgg//..1063//11222222 1/tp://11/221.1/oooogog.
[38] Z., Zhao, L., Zhang, Z., Zhang, H.. Liu, D., Liu, T., Metaxas, D.N.: Steering prototypes with
prompt-tuning for rehearsal-fir farersssl-feeeeeeeeeee continual learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2523-2533 2533 (January 2024)
[39] Lim, L.H., Sepulchre, R., Ye, K.: Geometric distance between positive definite matrices of different dimensions. IEEE Transactions on Information Theory 65(9), 5401-5405 (2019)
[40] Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., Tang, J.: P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 61-68. Association for Computational Linguistics, Dublin. Ireland (May 2022). https://do..rg//1..866555 wttp/8665//11122..eem 1/2022.acl-shorr. org/ 2022.0022..00 acl wcw.aaort.......... 8/
[41] Liu, Z., Zou, X., Hua, G., Zhou, J.: Token coordinated prompt attention is needed for visual prompting. arXiv preprint 2xXi:25505002000 (2025)
[42] Marks, M., Knott, M., Kondapaneni, N., Cole, E.,, Defraeye, T., Pérez-Cruz, F., Perona, P.: closer look at benchmarking self-supervised pre-training with image classification. CoRR abs/2407.12210 (2024), http:: htgg////oooooo//////
1t.///.//////////./ 480.0..8555aaaa http:/.xxiiv..... 2407 12210
[43] Matthey, L., Higgins I., Hassabis, D., Lerchner, A.: dsprites: Disentanglement testing sprites dataset. https//ggtaabb.ommmmemmmm//////e
(2017)
[44] Nawrot, P., Tworkowski, S., Tyrolski, M., Kaiser, E.,, Wu, Y., Szegedy, C., Michalewski, H.: Hierarchical transformers more efficient language models. arXiv preprint 1rX1v:2110.13711 (2021)
[45] Netzer, Y., Wang. T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natu- ral images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 (2011)
[46] Nilsback, M.E., Zisserman, A.: Automated flower classification large number of classes. In: 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. IEEE (2008)
[47] Niu, S., Miao, C., Chen, G., Wu, P., Zhao, P.: Test-time model adaptation with only for- ward passes. In: Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., Berkenkamp, F. (eds.) Proceedings of the 41st international Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 235, pp. 38298-38315. PMLR (21-27 Jul 2024), https::/pproceeddnng/ mlr prwss/eees...... v235/ 17235/niu220 niu24a......m html
press/v235
[48] Oymak, S., Rawat, A.S., Soltanolkotabi, M., Thrampoulidis, C.: On the role of attention in prompt-tuning. In: Proceedings of the 40th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 202, pp. 26724 26768. PMLR (23-29 Jul 2023), https::/procceedingg/ mlr prwpspressssee.. v202 wwwak23..a........o... html
[49] Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C.V.: Cats and dogs. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1498-3500 (2012). http:://doi.org/1.. 1ttp:///0.0.00000022220226..6600
[50] Paszke, A.: Pytorch: An imperative style, high-performance deep learning library. arXiv preprint
arXiv:1912.01703 (2019)
13

---

[51] Petrov, A., Torr, P.H.S., Bibi. A.: When do prompting and
work? theory of capabilities and limitations. International Conference on Learning Representations (2023). htips://doi.org/10.oo 405.0044...0/00.011..0....
[52] Rebuffi, S.A., Bilen, H., Vedaldi, A.: Learning multiple visual domains with residual adapters. Advances in neural information processing systems 30 (2017)
[53] Ren, L., Chen, C., Wang. L., Hua, K.: Da-vpt: Semantic-guides visual prompt tuning for vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4353-4363 (June 2025)
[54] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-basee localization. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (Oct 2017)
[55] Su, Y., Wang, X., Qin, Y., Chan, C.M., Lin, Y., Wang, H., Wen, K., Liu, 7., Li, P., Li, J., Hou, L.,, Sun, M., Zhou, J.: On transferability of prompt tuning for natural language processing, North American Chapter of the Association for Computational Linguistics (2021). https://oii.rr//I/ 18653// waw.0222naccmmmiin29599
[56] Van Horn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., Belongie, S.: Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)
[57] Veeling, B.S., Linmans, J., Winkens, J., Cohen, T., Welling, M.: Rotation equivariant enns for digital pathology. In: International Conference on Medical Image Computing and Computer- Assisted Intervention (2018)
[58] Vu, T., Lester, B., Constant, N., Al-Rfou', R., Cer, D.: SPoT: Better frozen model adaptation
through soft prompt transfer. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting the Association for Computational Linguisties (Volume Long Papers). pp. 5039-5059. Association for Computational Linguistics, Dublin. Ireland (May 2022). https:///oiiiooo 111.0/...66////2/2222222 acl.oon........oo............ httts::/aclanthhlllgg org/ 2022 acl-long 346/
[59] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.:
The caltech-ucsd birds-200-20 dataset. Tech. Rep. CNS-TR-2011-00) 1-0100 California Institute of Technology (2011)
[60] Walmer, M., Suri, S., Gupta, K., Shrivastava, A.: Teaching matters: Investigating the role of supervision in vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7486-7496 (2023)
[61] Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bow- man, S.: Superglue: A stickier benchmark for general-purpose language understanding systems. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Gar- nett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran As- sociates, Inc. (2019), https::/prroceeee
www.ips..eripp.eeee htcc//apr_ffileeee wwl.ffeeessaapae (eriiies/aaprr//// http:/appr///// / pdf
[62] Wang, J., Wang, C., Luo, F., Tan, C., Qiu. M., Yang, F., Shi. Q., Huang. S., Gao, M.: Towards
unified prompt tuning for few-shot text classification. In: Goldberg, Y., Kozareva, 7.,, Zhang, Y. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2022. pp. 524-
536. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Dec 2022). https://doi.org/100. http:ooorrg.0011131/111/1//1 116531111/022222222 12t.://0...iinnggemmm.3.7... http:at://aaaaaatllllloeo org/
2022. f fwndings-emnnp....... 37/
[63] Wang, P., Zheng, W. Chen, T., Wang, Z.: Anti-oversmoothing deep vision transformers via the fourier domain analysis: From theory to practice. In: International Conference on Learning Representations (2022), ptttps://ppereviiie
[64] Wang, S., Gao, J., Li, Z., Zhang, X., Hu, W.: A closer look at self-supervised lightweight vision transformers (2023)
14

---

[65] Wang. Y., Chauhan, J., Wang, W., Hsieh, C.J.: Universality and limitations of prompt tuning. Conference on Neural Information Processing Systems (2023), https //openreview mtt:fforrmmm/t-zzkk/////..
In:
[66] Wang. Y., Cheng, L., Fang, C., Zhang, D., Duan, M., Wang, M.: Revisiting the power of prompt for visual tuning. In: Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net (2024), https://openrevies http//orunnd-2293PP...
[67] Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning prompt for continual learning. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 139-149 (June 2022)
[68] Woolson, R.F.: Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials pp. 1-3 (2007)
[69] Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recog- nition from abbey to zoo. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 3485-3492 (2010). https://bb..og//l..119vcvvvv...
[70] Yoo. S., Kim, E., Jung. D., Lee, J., Yoon, S.H.: Improving visual prompt tuning for self-supervised vision transformers. International Conference on Machine Learning (2023). http::/do...oo//11..65552xxx2x12 485500/r2xw.23066.555 https://ooi.org/1..
[71] Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable features in deep neural networks? Advances in neural information processing systems 27 (2014)
[72] Zeng, R., Han, C., Wang, Q., Wu, C., Geng, T., Huangg, L., Wu, Y.N., Liu, D.: Visual fourier prompt tuning. Advances in Neural Information Processing Systems 37, 5552-555555555 2552-5555 (2024)
[73] Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucie, M., Djolonga, J., Pinto, A.S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., Houlsby, N.: A large-scale study representation learning with the visual task adaptation benchmark. ARXIV (2020)
[74] Zhang, X., Tian, Y., Xie, L.., Huang. W., Dai, Q. Ye, Q., Tian, Q.: Hivit: A simpler and more efficient design hierarchical vision transformer. In: The Eleventh International Conference on Learning Representations (2023), https: http:pppnnneeeeeeeeeeeerrrr////a
[75] Zhang. Y., Zhou, K., Liu, Z.: Neural prompt search, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
[76] Zhou, N., Chen, J., Huang, D.: ivpt: Improving task-relevant information sharing in visual prompt tuning by cross-layer dynamic connection. arXiv preprint arXiv: 2404.05207 (2024)
[77] Zhuo, L., Wang. Z., Fu, Y., Qian, T.: Prompt as free lunch: Enhancing diversity in source- free cross-domain few-shot learning through mmantic-gguided prompting. arXiv preprint arXiv:2412.00767 (2024)
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly indicate the scope and contribution of study in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Appendix A states the limitations of our work and possible room for improve-
ment.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
We don't 't include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We disclose every detail of our experiment in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All code and scripts necessary reproduce main experimental results are published https://ga httbb/ttttb...... com/ iam maekyun/vipana We comply with the NeurlPS code submission guidelines.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc//ppppp http:/cccccccgguueeeeeeeeeeeeeee wwaccc/guiies/eee httpubii//gi.iee/ddeeeaaaaiiaiia
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We discuss experiment details in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the standard deviation of our method's 's accuracy in Appendix F and G We also report the robustness of our method against randomness in Appendix I.7
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The computational resources, including GPU specifications used in our experi- ments, are detailed in Appendix D
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification:
Our research adheres the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In Appendix A state the broader impacts of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does not release any models or datasets with a high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the datasets, code, and models used are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the complete implementation of VIPAMIN, including compre- hensive documentation and hyperparameter configurations, as part of the supplementary materials.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Human subjects are not involved our research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Human subjects are not involved in our research
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is not involved in the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

Appendix Overview
The appendix section includes additional experimental results and discussions of our NeurIPS 2025 submission: VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion, summarized follows:
Section
A
discusses limitations and broader impact
Section
reviews related works depth
Section
presents the VIPAMIN algorithm and an overhead analysis
Section
elaborates experimental setup
Section
justifies the use of SPT/rand baseline
Section
reports full VTAB-1k results
Section reports full FGVC few-shot results
Section
studies adaptability of VIPAMIN, such generalization across models, scales,
and modality
Section provides further analyses, including GradCAM visualization, feature-selection ablations, hyperparameter roles, and analysis on supervised VPT
A Limitations and Broader Impact
Limitations While our method demonstrates significant improvements for self-supervised visual prompt tuning (VPT), several important questions remain open for future investigation. (1) Can the principles behind VIPAMIN generalize to other modalities, such as text or multimodal settings? (2) What are the fundamental representational differences between supervised and self-supervised pretrained models, and how do these differences affect prompt behavior? (3) Is there a principled approach for selecting the hyperparameters introduced in VIPAMIN (e.g.. k A), reduce reliance on empirical tuning? Addressing these questions would enhance the applicability and theoretical understanding of prompt-basel adaptation in broader contexts.
Broader Impact This work contributes to the advancement of parameter-efficient transfer learning by improving the reliability and adaptability of visual prompt tuning, particularly in self-supervised settings. By addressing key failure nodes-namely, uniform attention and representational collapse- method enhances the utility of VPT under distribution shift and in cats-constr regimes. These improvements may facilitate broader adoption of foundation models in domains where full fine-tuning is computationally prohibitive or impractical.
More broadly, the VPT paradigm represents shift toward more sustainable and accessible AI. By decoupling task adaptation from full model retraining, prompt tuning can significantly reduce the computational environmental costs associated with deploying large-scale vision models. This has the potential to democratize the use of powerful pretrained models, enabling resource-constrained institutions to more effectively leverage state-of-ther tfchh-rr-- vision systems.
B
Related Works
B.1 Detailed Review of Related Works
Theoretical Foundations of Prompt Tuning While prompt tuning has shown strong empirical performance, its underlying mechanisms remain incompletely understood. Recent theoretical work suggests that, under simplified conditions, gradient descent on prompt tokens can selectively modulate aecettin-amplilyiing focus on label-relevant tokens while suppressing noise [48). Other studies
underscore the expressive limitations of prompt-based methods. For example, [51 argue that
methods like prefix tuning introduce low-rank output biases without modifying attention patterns among input tokens, largely retrieving existing knowledge rather than enriching representations.
Our study extends these analyses by empirically examining sulf-supervised visual transformers under realistic training conditions. We find that randomly initialized prompts, commonly used in VPT, often fail to specialize or retrieve meaningful semantics during optimization. Moreover, the induced
23

---

low-rank bias frequently collapses into the subspace of the frozen self-attention output- suggessig that, in practice, prompt signals may be absorbed into the pretrained representation space, limiting their potential to guide adaptation.
Prompt Initialization Language and Vision Prompt tuning originated from hard prompts-
manually crafted natural language inputs-to steer pretrained language models (37] As the field shifted to continuous, trainable soft prompts, concerns arose around their optimization, particularly due to slow convergence under standard gradient descent [55]. This motivated prompt initialization strategies aimed at improving stability and performance. An early study by [36] compared Kavier-uniform initialization [17] to alternatives using frequent word embeddings and label-token embeddings. While these improved performance in smaller models, gains diminished with model scale. More recently, Pretrained Prompt Tuning (PPT) [9] showed that soft prompts perform well with ample data but struggle in few-shot settings due to poor initialization. PPT addresses this by pretraining prompts while keeping the backbone frozen, yielding strong improvements in data-scare, regimes.
Despite progress in the language domain, prompt initialization techniques have not been thoroughly explored in the vision domain. To our knowledge, the only prior work explicitly addressing visual prompt initialization is Self-Prompt Tuning (SPT) [66]. SPT observes that, during training, prompt representations tend to converge to the distribution of patch embeddings. Motivated by this, SPT proposes initializing prompts using prototype embeddings derived from clustering patch tokens, thereby accelerating convergence.
In contrast to SPT, our work takes fundamentally different approach. Rather than relying solely on clustering- based prototypes, we identify and address the two core limitations of visual prompt tuning- uniform attention and subspace collapse. Our proposed method, VIPAMIN, not only facilitates fast convergence and stable training through principled initialization but also enhances the representational diversity of prompts, enabling better adaptation under challenging conditions.
Prompt Tuning in Few-Shot Learning Few-shot learning is natural use case for prompt tuning, especially in resource-constrained settings. While most prior studies focus on NLP and multimodal domains [19] 35 62] 777,, they consistently report limitations in low-data regimes. PPT 19 was
among the first to identify poor performance in few-shot scenarios, attributing it to optimization challenges, and proposed prompt-only pretraining strategy to improve generalization.
In the vision domain, 75 similarly highlight the shortcomings of visual prompt tuning in extreme few-shot settings (e.g.. k- = 2), and introduce NOAH-a neural architecture search framework that integrates multiple parameter-efficient tuning (PEFT) methods to enhance few-shot performance.
B.2 In-Depth Comparison with Related Works
Self-Prompt Tuning (SPT) The underlying intuition behind Self-Prompt Tuning (SPT) is that the convergence of the prompt could be improved by initializing prompts using prototypical repre- sentations of patch tokens. However, this approach does not involve token-level selection; instead, it constructs prototypes without explicitly identifying which tokens carry semantically meaningful information. In contrast, VIPAMIN performs token-level selection in a scalable way and further enhances adaptation by injecting novel, orthogonal directions-an inductive bias that directly contrasts with SPT's prototype-based initialization. This combination proves more effective, especially under data-scarce scenarios.
Gated Prompt Tuning In [70]. the authors highlight that the expression of pretrained knowledge differs significantly between supervised and self-supervised models. Specifically, through the lens of deep image priors, they observe that the retention of image-specific information varies across transformer blocks, and the distribution of relevat features depends on the pretraining strategy. Our work complements these findings in two key ways. First, we identify and analyze specific failure modes of VPT self-supervised settings, offering a clearer view of how prompts interact with frozen embeddings. Second, we move beyond characterizing these differences to propose a practical method, VIPAMIN, that actively exploits the structural properties of self-supervised representations to improve adaptation.
24

---

SPARC (Continual Prompt Learning) Recent work in continual learning with prompts has proposed initializing new prompts the orthogonal subspace of previously learned ones, thereby facilitatingg adaptation to novel tasks while preserving prior knowledge. This approach, exemplified by SPARC, seeks to mitigate catastrophic forgetting by reusing task-specific prompts and conditionally initializing new prompts such that (P new: P ota) [26]. While this strategy bears similarity to our orthogonal initialization, the focus of our method is fundamentally different. Rather than emphasizing prompt reuse across tasks, we investigate the geometric relationship between prompt tokens and the representation space, and how this interaction evolves during training. In particular, we formally motivate the benefit of initializing prompts in the orthogonal complement of the dominant subspace of representations, highlighting its potential to enhance adaptation by directing attention toward underutilized directions.
CokkinCaodd Prompt Attention (TCPA) TCPA 41 addresses the limitation of shared prompts in VPT by assigning token- siken-specific prompts and modulating attention separately for CLS and patch tokens. This is achieved by explicitly masking and reweighting attention patterns throughout training. While VIPAMIN shares the motivation of improving prompt-token interaction, it takes different route: instead of modifying attention dynamics during training, it initializes prompts using token clusters from the pretrained key space, ensuring semantic alignment from the outset. More importantly, VIPAMIN explicitly mitigates prompt collapse into the representation subspace via orthogonalizatio. As shown in Table 4 and Figure S3] this proves particularly advantageous for dissimilar and few-shot tasks where prompt diversity and representational novelty are essential.
C
Algorithm
Detailed algorithm description for VIPAMIN is introduced in Algorithm I
Algorithm VIPAMIN
Input: Minibatch B, Pretrained Vision Transformer M with parameter pore, Locality factor of the matching module k, Relative strength of the orthogonalizing module X, Prompt length Np.
Randomly initialized prompts Prand = {prand prand ,PN, .NN.d
Set of generated prompts Pu {p1, P2, ,N,, > Input Preparation
Output:
Fetch self-attention weights of the first block, (Wq. W k. Wv. bo. bx. bv C Gprel Fetch Eg == PosEmbed + PatchEmbed(B) via feedforwarding B through M
Fetch SA SA(E)) + softmax (EE, + b2) (E, Wx bx)² /vd) (Eo Wy + bw)
Matching Module
IndexMap = RowWiseTopKIIdd.ces (Prand x EoWx T KW R)
for i + to N, do
p matchh +- RowWiseMean E, [aplnexxMa,,
end for Pmaach (pgaatch, p pmaatch....
-NN, immatok)z
Orthogonalizing Module SVD Decompose SA(Es), UEVT = SA(Ea) i to N₂ do porth + (I (I- (p, and Wv +vbbr W b
end for
Porth + (p porth PN porthy?
Po + (1 A)Pgmatch + APorth
Perform prompt tuning with prompt-prepended input, (Po: Eo]
I For models which don't employ biases the self-attention module, bas bg, by are set to zero vectors.
AA diag (||A,115_)) A
25

---

Table S1: Computational overhead analysis on Flower-102. MoCo-v3 pretrained model used to
report the accuracy.
Tuned/Total
Additional FLOPss
Init. GFLOPs
Accuracy
VPT
0.30%
(17.7 GFLOPs)
0
90.2
0.20%
<0
0
91.3
iVPT VEPT
0.31%
IM
N/A
0.30%
5.5M
0
92.4
SPT-Deep 0.30%
0
3300
93.2
VIPAMIN-Deep
0.30%
0
1.73
94.0
C.1 Overhead Analysis
The computational overhead introduced by our method is minimal. We execute each module 5 times using model ViT-B/16, prompt length of 100, and training batch from Resisc45, Excluding the time required for model and dataset loading (denoted as Input Preparation in Algorithm [ the average wall-clock time to execute the core logic of the matching module is 3.94 seconds, with a standard deviation of 0.06 seconds. The orthogonalizing module incurs slightly greater overhead due to its reliance on singular value decomposition and eseudo-inverss computation; its core logic requires 9.14 seconds on average, with standard deviation of 0.11 seconds. Importantly, this overhead is incurred only once during initialization and is subsequently amortized over the full training process.
Continuing the comparison with other VPT-Deep variants presented in Section 5.3 we further substantiate the superior efficiency of VIPAMIN-Deep in multiple dimensions. Table SI shows the result. First, VIPAMIN, along with SPT-Deep, introduces no additional parameters to be tuned and no extra FLOPs relative vanilla VPT. Furthermore, for one-time initialization, SPT incurs an overhead that is orders of magnitude larger than ours. In addition, the total additional runtime of VIPAMIN over VPT is under 30 seconds, and the memory overhead is modest the peak GPU allocation 2.2 GB on an NVIDIA A6000. Lastly, the seemingly superior efficiency of E² EPV is largely due to its pruning module, which entails a two-stage training procedure. Overall, VIPAMIN-Dece delivers strongest accuracy-compute trade-off among the VPT-Deep variants.
D Experimental Details
Benchmark Datasets Specification this section, we provide additional details on the benchmarks used for evaluation: VTAB-IK and FGVC. The 19 datasets included in VTAB-IK are summarized in Table S2 For the FGVC few-shot experiments, we sample from the training set and use the validation set for both hyperparameter tuning and extracting embeddings for the initialization procedures of SPT/rand and VIPAMIN. Additional benchmark specifications are also provided in Table S3
Hyperparameter Specification Our hyperparameter settings largely follow the configuration used in [66]. However, for self-supervised backbones, we narrow the learning rate range based the recommendations from 77) The full set of hyperparameters is detailed in Table S4
Hyperparameter Tuning and Evaluation Protocol We perform a grid search over a predefined hyperparameter pool the training and select the configuration that achieves the highest validation accuracy. The final reported performance corresponds to the average top-1 accuracy on the test set, computed over three independent runs using the selected hyperparameters. The complete tuning pool is provided in Table S4
Training Configurations We use the AdamW optimizer with a batch size of 32. The learning rate follows a cosine decay schedule with a linear warm-up period of 10 epochs, consistent with prior work 666 Unless otherwise specified, we use prompt length of 100.
Reproducibility The implementation of VIPAMIN is based on PyTorch 50
with VTAB-1k
datasets loaded via TensorFlow Datasets (., following established practices in 20 experiments were conducted on NVIDIA A6000 GPUs with 40GB of memory.
66
70
All
26

---

Table S2: Specifications of the VTAB-Ik [73 benchmark.
Dataset
Description
# Classes
Train
Val
Test
Preprocessing
CIFAR-100
10,000
1o 14
6,084
DTD -
47
1,880
Flowers 02 46
Natural
102
800/1000
200
6,149
Pers 39
37
3,669
SVHN 45
10
26.032
Sun397 69
397
21,750
Patch Camelyon
57
32,768
EuroSAT 23
Specialized
10
800/1000
200
5,400
Normalize by ImageNNtt Statistics
Resisc45
g
45
6,300
Retinopathy
A
5
42,670
Clevricount
15,000
Clevridistance
28
15,000
DMLab
2
22,735
KITTV/distnnce
16
711
Structured
800/1000
200
dSprites/locatin
4
16
73,728
dSprites/orientation
4
16
73,728
Bazziuh
34
18
12,150
SmallINORB/elevationn
34
9
12,150
Table S3: Specifications of the few-shot datasets from FGVC. : Due to class imbalance, the training set size falls short of Class Number
Dutuset
Description
Clasess
Train size by ivshot
Val
Test
Augmentation
k
k
NABirds 56
CUB-200-20U1
59
Bird species reangaition
Bird species recognition
555
555
200
1,110
400
2,220
300
4,427'
1,600
2,393
600
24,633
5,794
-+ Random crop to 224 224
to
Oxford
species recognition
204
408
Horizontal
Stanford Dogs
Stanford Cars
60
Dog breed recognition
Car model classification
196
120
196
240
392
784
480
1,568
960
1,200
815
8.580
8.041
Normaliz (ImageNet stats)
E
Justification on SPT/rand
Although VIPAMIN outperforms the T-Meens-based SPT results reported in the original paper VTAB-IK lk, reproducing this initialization computationally prohibitive for the full set of experiments conduct. As alternative, adopt SPTT/rnd, which achieves the best performance among SPT variants according to Table 4(b) in [66] However, as the reported results for SPT/rand are limited to only datasets, compare reproduced SPT/rand results with the original SPT accuracy VTAB-1K. The result can found in Table
We find that, with the MoCo-v3 backbone. SPT/rand performs comparably to the original SPT. In contrast, under the MAE-pretrained backbone, SPT/rand underperforms by approximately 2% in mean accuracy. To ensure consistency with the reported results and to maintain fidelity in baseline comparisons, we conduct our primary experiments using the MoCo-v3 pretrained model.
F Full Results on VTAB-1k
Table S6 presents the top-1 accuracy for full fine-tuning and VIPAMIN across the 19 individual datasets VTAB-Ik. For VIPAMIN, report both the mean and standard deviation of top-1 accuracy, computed over three independent runs with different random seeds. When evaluated on MoCo-v3 pretrained models, VIPAMIN (shallow or deep) outperforms full fine-tuning 16 out of 19 tasks. For MAE-prette Ereerrrtrd models, it outperforms full fine-tuning in 14 out of 19 tasks. While deep variant generally exhibits stronger performance than the shallow variant-likely due to increased representational capacity from higher prompt dimensionality-thee (e.g., Clevr/count on MAE, DMLaa on MoCo-v3). Overall, these results demonstrate that VIPAMIN remains effective even when extended to deep prompt configurations.
are few exceptions
27

---

Table S4: Hyperparameter tuning range
SP
VIPAMIN
Batch size
32
Learning rate scheduler
Cosine Decay with Lincar Warmup
Total epochs
100
Prompt length
Optimizer
AdamW
Learning rate Weight decay
f0.05, 0.1, 0.25, 0.5, 1.0, 5.5,
A
{2, 32, 128) (0.0, 1.0]
À
Table S5: Per-task fine-tuning for VTAB-IK Ik benchmarks with pretrained VIT-B/16 as backbone. Bold indicates the better method between SPT and SPT/rand for each task.
Natural
Specialized (4)
Structured (8)
Patch Camelyon
Clevridistance
KIFTiidistane
SmalINORB/an
SmallN ORBIIO
Caltech101
CIFAR- 100
Flowers102
Retinopatiy
Clevriountt
dSprites/loc
dSpriess/nn
SVHN
San399
EunSAT
Resiso4s
DMLab
DTD
Pets
Mean
Methods
MTBB with MoCo-v3 pretroiied on ImageNNtt11K
SPT/rand
89.8
59.3
87.9
83.7 39.7
83.2 94.6
84.5
71.3
56.4
59.5
79.9
75.7
48.4
29.7
46.5
68.3
91.0
58.1
91.1
82.2 39,9
83.6
75.4
73.3
60.6
71.4
75.0
42.1
28.0
45.2
68.3
WTBB with MAE pretnained ImugeNet--1k IK
SPT/aand
83.8
25.2
57.7
72.2
71.2
76.0
20.8
77.4
91.9
73.0
73.3
73.6
61.7
40.5
75.0
72.8
44.6
27.9
33.0
60.6
SPT
87.6 29.5 61.5
77.4 80.8
77.2
23.7
84.4 93.0
70.8
75.4
73.3
55.5
44.0
73.2
70.6
48.0
27.4
35.7
62.6
G Full Results on FGVC Few-Shot Experiment
We report the full top-1 accuracy results, including standard deviations over three independent runs with different random seeds, for our few-shot experiments on the FGVC benchmark. This includes the shallow and deep variants of VPT, SPT/rand, and VIPAMIN across both MoCo-v3 and MAE pretrained models. Results for the MoCo-v3 pretrained models summarized in Table SS7 Among the 20 configurations evaluated (five datasets across four shot counts), VIPAMIN- achieves the highest accuracy in 14 cases. Moreover, when averaged across the five datasets, VIPAMIN-Deep consistently outperforms all baselines at every shot level, underscoring its robustness and adaptability in data-scarce regimes.
comparable trend is observed for MAE-pretrained models, as reported Table Here, VIPAMIN- Shallow obtains the best performance in 5 cases, while VIPAMIN- Deep achieves the highest accuracy cases, totaling out of 20. terms of mean accuracy across datasets, VIPAMIN-Deep again demonstrates superior performance at all shot levels.
These results collectively reinforce the effectiveness of VIPAMIN in alleviating the pronounced performance degradation typically observed in prompt tuning on self-supervised backbones under few-shot learning conditions. Its deep variant, in particular, exhibits consistent and substantial improvements over existing baselines, highlighting its capability to generalize across both pretraining paradigms and data scarcity scenarios.
H Adaptability of VIPAMIN
To examine how well our method works beyond pretrain-then-fieeume image classification, we conducted multiple experiments VIPAMIN under challenging conditions. These experiments included variations in model sizes, pretraining strategies, out-of-distribution generalization, and modality.
28

---

Table S6: Per-dataset performance comparison between Full fine-tuning result, VIPAMIN -Sallloww and VIPAMIN-Dege on VTAB-Ik with MoCo-v3 ViT-B/16 and MAE ViT-B/16, over three indepen- dent runs. "t" indicates results reported [66].
Dataset
MoCo-v3 VIT-B/16
MAE ViT-B/16
Full
VIPAMIN Shallow
VIPAMIN Deep
Full
VIPAMIN Shallow
VIPAMIN
Deep
Natural
Caltech101
91.0
92.21
-0.24
91.05
84.2
88. 48.8222
89.01 =0.52
CIFAR-100
57.6
71.34 10.30
73.01 10.15
24.6
35.95 10.52
24.82 114.28
DTD
64.6
69.96 20.22
68.85 10.50
56.9
58.92
59.95 40.50
Flowers
91.6
92,40 +0.29
43.99
72.7
74,4
75.53 =0.43
Pets
79.9
87.79 10.13
87.97 10.04
74.4
71.88 11.03
79.53 10.10
SVHN
89.8
86.53
10.06
86.83 29.24
86.6
80.04
81.83 =1.80
Sun397
29.1
10.12 37.04
42.13 10.02
15.8
23.05 10.12
23.17 10.40
Specialized
Patch Camelyon
85.1
82.83 H0.26
85.89 :0.54
81.8
80.13
84.24
EuroSAT
96.4
96.26
96.61 10.14
94.0
92.78 10.53
93.52 =2:52
Resisc45
83.1
83.07
10.24
84.80 10.16
72.3
73.26 10.12
75.23 10.33
Retinopathy
74.2
74.39 +0.20
71.84 20.73
70.6
73.66
68.66 84
Structured
Clevs/count
55.2
69.23
10.72
74.65 10.95
67.0
73.65 10.42
70.61 13.24
Clevr/distance
56.9
59
49:62
63.54 =0.70
59.8
67.62
56.82.2356
DML.ah
44.6
47.56 19.14
47. 10.55
45.2
43.37
44.42
KITTI/distance
77.9
78.25
79.05 #0.60
75.3
78.06
80.03 -1.22
dSprites/loc
63.8
75.82 10.84
84.49 12.41
72.5
80.02 11.96
82.26 :1.32
dSprites/ori
49.0
47.81 11.61
47.15
47.5
46.96
49.50 =0.35
SmallNORB/azi
31.5
26.74 10.31
29.27
10.35
30.2
26,92 10.66
23.23 10.20
SmallNORB/elle
36.9
48.74
45.17 49.47
33.0
53.13
46.39 =0.97
H.1
Robustness under Distribution Shift
To test whether our approach is confined to the standard pretrain-then-ieee paradigm, we evaluate Out-of-Distribution (OOD) generalization, stringent check of robustness beyond in-distribution tuning. Specifically, we corrupt the CIFAR- 100 test set with multiple distribution shifts and compare against baselines. As summarized in Table S9 method attains the best accuracy 5 of 6 corruptions and the highest mean accuracy, improving by 4.0% over SPT/rand. These results indicate that the benefits of method extend beyond the pretrain-then-finetue setting, translating into stronger performance under distribution shift.
H.2 Generalization to Self-Supervised Models and Scales
While main paper primarily evaluates VIPAMIN using MAE and MoCo-v3 with ViT-Base and larger models, we further investigate its adaptability across different self-supervised pretraining paradigms and model scales. To this end, we experiment with DINO ViT-Small/16 and MAE ViT- Tiny/16, representing the regimes of contrastive learning and masked image modeling, respectively (5) 64
DINO distinguishes itself from MoCo-v3 by employing self-distillation via momentum-updatted teacher, while still retaining contrastive training structure. Notably, DINO achieves 81.5% Top-1 accuracy on ImageNet with ViT-S/16 under fine-tuning. On the other hand, MAE-Tiny, although not included in the original MAE paper 22), has been adapted in [64], which introduces tailored pretraining strategy for smaller architectures. This setup yields 78.0% Top-1 accuracy on ImageNet with ViT-Tiny.
Table presents the results on VTAB-1k. For DINO ViT-Small/16, VIPAMIN outperforms full fine- tuning for 16 out of 19 tasks, highlighting its strong adaptability to smaller architectures. However, in

---

Table S7: MoCo-v3 few-shot accuracy (%) for k-shot classification across FGVC datasets over three
independent runs.
k-shot Method
Mode
CUB
NABirds
Flowers
Dogs
Cars
Mean
Shallow
13.67/0.34
7.6810.35
31.4211.15
31.18:2.41
4.7410.10
18.14
VPT
Deep
16.87+0.06
8.96#0.39
36.51-21.74
33.87-0.05
4.60+0.15
20.16
Shallow
17.16:0.31
11.67:0.42
48, 48.90
35.52-0.31
5,29:0.u2
23.71
SPT/rand
Deep
21.37.0.06
13.46:0.20
53.07:0.28
36.41:0.25
5.70:0.07
26.00
Shallow
20.1410.12
12.61:0.08
52.81:n.05
37.47.0040
5.7310.14
25.75
VIPAMIN
Deep
21.65+0.13
13,65:0.08
53.66H0.32
36,89=0.21
5.92+0.13
26.35
Shallow
15.59110.07
11.6910.21
59.031194
45.40..173
6,40:9.12
27.62
VPT
Deep
38.1911.67
7.4049.06
32.22:0:0
34.50123.56
9.20H1.38
32.30
Shallow
29.8.24032
22.54:0.48
70.44:122
48.97.0.1
10,90H0.05
36.54
SPT/rand
Deep
38.6540.16
26.13:0.10
75.00+0.29
50.1 5=0.00
11.78#0.09
40.34
Shallow
35.97:0.43
23.14:0.22
71.62:021
49. 39H0.12
11.06H0007
38 24
VIPAMIN
Deep
38.5310.19
26.92:0.09
75.42:0.09
50.86.0.19
12.05.0.11
40.76
Shallow
31.3550.55
14.29:0044
66. 10
36.8
9.88+0.14
31.71
VPT
Deep
57.71:0.17
29.28:0.56
83.69.109
63.06:0.15
8.981787
48.54
Shallow
51.7310.74
40,46:0.13
84.64
09.844..26
21.6610.15
51.67
SPT/rand
Deep
58.10.
44.81+0.1
84,48
60.74=0.12
24. 1240.07
54.45
Shallow
53.6510.74
40,9610.09
85. 19
60.25:0.15
21.8810.11
52.39
VIPAMIN
Deep
58.0140.12
44.92:0.05
85.79.127
61.27:0017
25.55:0.05
55.11
Shallow
37.25:n:22
17.22::014
77.7910.39
62.7949.23
13.46H0.4
41.70
VPT
Deep
62.7810.75
31.24:0.15
89.48:0.01
55.95
26.33:15.97
53.16
8
SPT/rand
Shallow
66.5840.27
54,9620.21
92.9000.04
69.1420.07
43.80H0.32
65,45
Deep
70.11:0.13
18.4019.11
94.922n.11
70.25:0.08
48.6410.32
68.46
Shallow
68.554n.22
52.12:0.06
94.3410.08
70.03:0.00
43.82HD.13
66.37
VIPAMIN
Deep
69.6340.14
58.68:0.09
95.51:0.15
69.66-9.24
51.01
68.90
Table S8: MAE Few-shot accuracy (%) for k-shot classification across FGVC datasets over three independent runs.
K-shot
Method
Mode
CUB
NABirds
Flowers
Dogs
Cars
Mean
Shallow
2.2810.06
0.801D.02
13.2613.51
1,081.133
0.9610.13
3.68
VPT
Deep
2.03:0.88
0.78
12.59
2.55:0.22
1.6110.99
3.91
Shallow
3.65:0.04
1.67:0004
30.0410.29
7.27:9.24
2.9020.08
9.91
SPT/rand
Deep
4.07:0.09
2.15:0.10
29.10H0.72
10.21:0.09
2.79H0.09
9.66
Shallow
1.400.
1.90g0.18
30.56H0.15
8.2942003
2.95:0.09
9.97
VIPAMIN
Deep
4.55
2.21:0007
30.3510.45
11.32:0.26
2.8310.10
10.65
Shallow
3.80H0.2
2.26:0
28.1GH431
8.40%1.77
3.30+0
9.38
VPT
Deep
3.6410.6
2.40
32.1740.51
6.21-9975
3.6240.05
9.61
2
SPT/rand
Shallow
29
4.40Ho.11
46.02H0.91
16,43:0.32
5.54.0.13
15.72
Deep
7.38+0.00
4.70H0.07
46.384041
23..889.29
5.21+0.77
17.39
Shallow
7.8540.49
4.84H:.34
46.53H 66
17.40=0.21
5.2140225
16.37
VIPAMIN
Deep
8.30+0.16
4.61=0.12
48.43+0.50
23.57-0.34
5.84
18.15
Shallow
9.12999
5.2220.16
46.70 74
23.2810.46
4.841046
17.43
VPT
Deep
12.77:0.52
11.1340.3
19H1.61
18.30
8.1244
10.10
Shallow
17.32:0.51
13.19
67.42H0.13
31.21:0.37
8.7411.18
27.18
4
SPT/rand
Deep
18.9110.48
14.3810.11
66.9910.44
39.45
9.3710.37
29.82
Shallow
18.8210.32
14.49.0.29
67.32.0.91
35. 35.2910.68
8.77.0.04
28.14
VIPAMIN
Deep
17.9919.28
14.2810.22
69.20H0.12
38.9319.44
9.78
30.04
Shallow
14.532135
10.70
63.0240.9
28.68115.84
8.3 40
25.45
VPT
Deep
18. 88.800
22.71
76.6340.89
16.00-5.20
20.31+0.30
30.49
Shallow
37.3120.70
31.30=0.0
80. 72:0.4
49 49.9..
25.4220.0 89
44.05
8
SPT/rand
Deep
40.50:0.12
36.1240:43
82.6 82.6140.59
53. 33.66-0.11
25. 60090.44
47,30
Shallow
37.55H0.78
31.7720.38
82.05H0.57
33.2
28.54:0.39
46.22
VIPAMIN
Deep
39.2240.1
34.224036
84.1140.21
53,9820.38
27. 27.1339.00
47.33
the case of the smallest model, MAE ViT-Tiny/16. VIPAMIN falls short of full fine-tuning. While VIPAMIN still achieves substantial improvements over VPT on MAE ViT-Tiny/16 (as shown later in Figure S1) its relative underperformance in this low-capacity regime remains open question.
We hypothesize that the limited embedding dimensionality-for instance, only 16 dimensions per head in the 12-head ViT-Tiny/16 nedet-restricts representational flexibility. In such settings, the injection of prompt tokens may dilute critical interactions among patch embeddings, ultimately leading to suboptimal performance.
30

---

Table S9: OOD Generalization test on CIFAR-100, Used ViT-B model pretrained by MoCo-v3.
Brightness Contrast Rotation Gaussian Shot Speckle Mean
VPT SPT/rand VIPAMIN
36.2
20.9
6.80
13.2
21.4
25.5
20.6
65.5
47.2
16.8
20.2
37.0
44.5
38.5
71.4
57.5
18.7
19.8
39.9
48.8
42.5
Furthermore, we investigate the extension of VIPAMIN to hierarchical transformers-vision backbones that build multi-scale features via staged downsampling (a pyramid of token resolutions) rather than a single, fixed-resolution sequence. This structure enhances efficiency and is standard in tasks such object detection [4]. We instantiate this with HiVIT (74), selected for its explicit pairing of hierarchical transformers with masked-image modeling (MIM) pretraining. Unlike typical hierarchical ViTs, HiViT drops the windowed attention and patch merging stage. As shown in Table STT VIPAMIN outperforms baselines on 12 of 19 tasks, demonstrating effective transfer to hierarchical backbones and highlighting its adaptability.
Table S10: Per-dataset performance comparison between Full fine-tuning and on VTAB-Ik with DINO ViT UTT-Smal//16 16 and MAE ViT-Tiny/11, over three independent runs. For full fine-tuning, search learning rate from 10.0001, 0.0005, 0.001, 0.005] following (27).
DINO ViT-Small/16
MAE ViT-Tiny/16
Dataset
Full
VIPAMIN
Full
VIPAMIN
Natural
Caltech101
73.41 11.32
83.01 11.50
61.42 11.37
51.69 10.33
CIFAR- 100
22.32
51.13
+.251
10.09
DTD
54.24 10.82
10.55 60.44
38.58
31.22
Flowers102
80.20
84.28 10.70
55.78
46.60
Pets
54.
82.24
2..8
20.19
SVHN
84.32 10.68
58.85 127.32
84.29 10.57
69.30 11.21
Sun397
14.98 +11000 .2.02
28.81 40.40
40.6
8.73 40.23
Specialized
Patch Camelyon
82.04
78 08.06+1199 197
78.69 +1.09
75.51
+0.83
EuroSAT
95.61 10.51
92.61 10.44
92.17 10.20
88.57 10.38
Resise45
77.30
75.44
61.96 10.26
53.39 10.48
Retinopathy
72.65
73.36 40.24
70.98
71.81
40.42
Structured
Clevr/count
43.31 +0.99
70.73 +0.50
62.67 +1.68
62.21 +0.43
Clevr/distance
49.42
61.51 10.46
60.40 +1.29
DMLab
40.50
10.57 44.49
40.96 10.78
37.19 10,80
KITTI/distance
77.68 #0.86
70.37 50.57
71.82+2.1
65.07 +2.20
dSprites/loc
46.48 +1
73.85
+0.45
78.88 10.36
76. 40.79
dSprites/ori
37.24
40.92
12.24
37.03 40.37
34.17 10.95
BmallNORB/azi
24.82 11.63
20.98 10.09
27.44 10.84
23.25 10.57
SmallNORB/ele
29.60+0.92
34.42
40.21
45.21 40.52
H.3
Transferability to Language Tasks
While our motivation in the main paper is limited to visual tasks, we also test our method in the language domain. We apply our method BERT-Large (IT] on the SuperGLUE benchmark [6]] to test generality across modality and scale. SuperGLUE is an extensive benchmark including the following masks-natural language inference (RTE, CB), coreference resolution (WSC), sentence completion (COPA), word sense disambiguation (WiC), and question answering (MultiRC, ReCoRD, BoolQ), As shown in Table S12 our method outperforms both P-Tuning v2 [40] and full fine-tuning, two of the strongest baselines in the language domain. This is intriguing result, as it suggests that our motivation regarding selective token attention and the injection of new knowledge can naturally extend to the language domain as well.
31

---

Table S11: Per-dataset performance comparison between baselines and VIPAMIN(-Shalloo) on VTAB-IK 1k with HiVIT, over three independent runs.
Dataset
VPT
SPT/rand
VIPAMIN
Natural
10
88.26 10.33
88,33
10.20
88.35 10.38
CIFAR-100 --0
40.41 #0.96
40.45
=1.34
41.05
#0.76
DTD
59.98 10.18
60.00
10.22
60.50
10.72
102
74.65 +0.78
76.44 =0.08
76.33 #0.68
Pets
76.71
76.91 =0.41
76.47
SVHI
75.61
81.58 10.61
82.60
Sun397
22.18 +0.43
22.14
22.33 #0.30
Specialized
Patch Camelyon
78.44 10.36
78.40 10.36
79,52 11.10
EuroSAT AT
10.12
94.65
#0 30
Resisc45
68.96 19.72
20.57 70.40
70.22 10.77
Retinopathy
73.80 10.05
74.44 10.21
74.28 10.37
Structured
Clevr/count
69.46=2024
Clevr/distance
62.59
61.65 10.36
62.23
DMLab
45.12
45.36
46.16
KITTI/distance
79.93
81.62
82.23 29.29
dSprites/loc
83.72 10.72
82.94
86,49 10:46
dSprites/ori
25.79 +2.78
29.32
51.44 #0.88
BmallNORB/azi
21.50
24.21 +0.30
22.61
#0 18
SmalINOORB/ele
39.53 10.48
41,20 10.32
44.02 11.42
Table S12: Per-task results for SuperGLUE development set with pretrained BERT-Large. 45n indicates results reported in 72
BERT-L (335M)
BoolQ
CB
COPA
MultiRC ReCoRD
RTE
WIC
WSC
Mean
Full
77.7
94.6
69.0
70.5
70.6
70.4
74.9
68.3
74.5
Prompt Tuning"
67.2
80.4
55.0
59.6
44.2
53.5
63.0
64.4
60.9
P-Tuning v2
73.1
94.6
73.0
72.8
78.3
75.1
75.7
BVPPT*
74.4
80.4
77.0
65.8
71.9
78.7
74.3
67.3
73.7
VEPT
74.8
81.2
78.1
67.8
72.9
77.2
75.3
68.4
74.5
VIPAMIN-Decp
74.6
94.6
79.0
66.4
70.6
79.1
74.3
69.2
76.0
Further Analyses
L. Extensive Comparison with VPT
Continuing from Figure ((b) of our main paper, report the impact of VIPAMIN in terms of LP ratio (i.e., proxy for task similarity to the pretraining task), and how effectively counteracts the underperformance of vanilla VPT.
Figure ST presents the results. Two key observations emerge. First, the underperformance of VPT on tasks with low LP ratios is a consistent trend across different self-supervised pretraining strategies and model scales. Second, while VIPAMIN consistently improves upon VPT across all LP ratio regimes, its gains are particularly pronounced on tasks with lower LP ratios. This suggests that addressing uniform attention patterns and prompt subspace collapse is crucial for enhancing the adaptability of VPT, especially in tasks that exhibit significant distributional shifts from the pretraining domain.
1.2 GradCAM Analysis
For interpretability, we visualize the GradCAM maps of the trained models various datasets. Results are shown in Figure[S2] Two distinct failure modes of VPT become apparent, each manifesting differently depending on the task similarity to the pretraining distribution. For tasks that are similar the pretraining domain, VPT exhibits overly uniform attention across the image, failing to localize semantically meaningful regions. In contrast, VIPAMIN produces sharper and more localized attention maps, owing to its matching module, which encourages prompt specialization through semantically coherent token alignment.
32

---

+39.1%
+15.7%
-0.2%
+60.8%
+40.6%
+23.0%
VPT/FT- (9%)
VPT/FT 1 (%))
-20
-20
40-
40
VIPAMIN
660
VPT
-20
VPT
Dissimilar
Similar
Dissimilar
Similar
VTAB (LP/FT)
(LP/FT)
(a) MoCo VTT (TT-Base16
(b) MAE VIT- Basc/1s
100
+
+49.9%
+38.0%
+18.8%
+%3.2%
+15.2%
+14.2%
VPT//F 1 (%)
VPT/FT 1 (96
25
-25
50
VIPAMIN
-00-
VIPAMIN
VPT
VPT
Dissimilar
Similar
Dissimilar
Similar
Sorted VTAB Datasets (LP/FT)
Sorted VTAB Datasets (LP/FT)
(c) DINO ViT-Small/16
(d) MAE ViT-Tiny/16
Figure S1: Relative accuracy ratio of VPT and VIPAMIN compared to full fine-tuning across 19 VTAB datasets, sorted by the LP ratio. The three percentages highlighted in red at the top of the figure denote the mean improvement VIPAMIN over VPT within each of the three LP ratio segments-low, medium, and high-as as defined by the tertile split of the dataset ordering.
On dissimilar tasks, VPT often fails to grasp the task semantics, resulting in attention focused on irrelevant background regions. This suggests inability adaptively reconfigure the prompt representation. VIPAMIN overcomes this limitation by explicitly injecting novel representational directions into the prompt space via its orthogonalizing module. As result, VIPAMIN consistently attends to task-relevant regions, demonstrating improved interpretability and alignment with human intuition.
Similar Tasks
Dissimilar Tasks
VPT
VIPAMIN
VPT
VIPAMIN
Figure S2: GradCAM visualization result of DINO ViT-Small/16 on various VTAB-Ik datasets.
L.3 Feature Selection in Matching Module
The matching module in VIPAMIN relies on key-space refined features (Zg W K) compute similar- ity between prompts and input tokens. To assess the impact of the feature type used for matching,
33

---

conduct an ablation study comparing various reference features extracted from the first transformer block:
Lo W Zo K3
SA(Zo)
softmax
W
Vd
Z B1 EZ)
When using the attention weights (blue box) as a reference, we directly apply the softmax scores for patch selection. other feature options (black boxes), we compute cosine similarity to identify the top-k tokens. As shown in Table [T3 using Zo Wr к yields the best performance, while attention- based selection performs the worst. This suggests that explicitly computing similarity shared projected space is critical to the effectivvness of VIPAMIN.
Interestingly, this observation aligns with findings from token merging studies in vision transform- ers [3] which aim to accelerate inference by merging redundant tokens. In that context, key vectors (Eo W k) have been shown effectively summarize token content for measuring redundancy. Simi- larly, our use of the key space exploits the natural clustering of semantically related tokens, enabling more coherent and task relevaett prompt initialization.
Table S13: Ablation on feature selection. MoCo-v3 pretrained model is trained and evaluated on Specialized tasks in VTAB- 1k. Hyperparameter set to zero for clarity.
Feature Mcan Ace
Ze
82.36
Ze W Q
82.39
Zo W x
82.85
softmax
81.80
Zo Wv
82.23
Z1
82.22
L.4 Role of Hyperparameters in VIPAMIN
Extending the analysis from Section 5.4] we investigate the functional roles of VIPAMIN's two core components by examining trends in optimal hyperparameter settings across tasks. VIPAMIN introduces two key hyperparameter: k, which specifies the number of tokens which each prompt attends, and X, which controls the degree of orthogonal bias injected into the prompt initialization.
Figure S3 (a) and (b) present results from VTAB-IK Ik and FGVC benchmarks, respectively. For tasks that exhibit significant distributional shift from the pretraining domain, smaller values of k and approaching yield the best performance. This suggests that such tasks benefit from more localized attention and stronger orthogonalization, which together enable the model to focus on task-specific features not captured by the pretrained backbone. In contrast, tasks more closely aligned with the pretraining distribution tend to favor larger & and smaller X, indicating that broader attention and greater reliance on pretrained representations are advantageous.
In few-shot scenarios, similar trends are observed for as the number of shots increases, implying that orthogonal bias provides useful inductive structure when prompt parameters cannot be fully optimized due to limited supervision. While no definitive trend is observed for & in the few-shot setting, larger values generally lead to robust performance across varying shot counts.
1.5 Extension toward VPT-Deep
Empirically, demonstrate that VIPAMIN yields performance gains even under the VPT-Deep variant as shown in Sec. [.3) raising the question of whether prompt subspace collapse persists in deeper architectures. To check this, for each transformer block / [L]. with input [Pi-1; X,-1), compute the projection energy, as defined in (
ProjectionEnergy (PP-WWY (X))))))))",
(1I)
34

---

k 8
k =
Normalized Accuracy
0.8
Normaliize Accura
8.
k 32
k 32
k 128
0.6
k= 128
0.6
004
00
0.5
A= 0.5
A= = 1.0
1=100
Dissimilar
Similar Sorted VTAB Datasets (LP Ratio)
1
Number Shots
(b)
Figure S3: Efffet Hyperparameter (a) We evaluate the performance of MAE pretrained VIPAMIN on VTAB-1K, across varying values of k, X. Datasets are sorted by their LP ratio (i.e., the relative closeness of linear probing to full fine-tuning performance) from lowest to highest. (b) We evaluate the few-shot performance of MoCo-v3 pretrained VIPAMIN NABirds. The z-axis represents the number of shots. For both figures (a) and (b), we select the best learning rate based on peak validation accuracy, and normalize the resulting accuracy to [0, range dataset (column). We present rough trend line of best performing hyperparameters red arrow.
VPT
VIPAMIN
1.0
Projection energy
0.8
0.6
0.4
shallow
deep
1
shallow
deep
0.2
0
25
50
75
100
25
50
75
100
Epoch
Figure S4: Deep projection energy trend during training dSprites/loc-the most dissimilar task.
where W denotes the value projection matrix in the /-th block and SA₁(-) is the self-attention module of 1-th block. As shown Figure S4 prompt collapse observed only in the first layer-where the initial shallow prompt is prepended -and does not occur in subsequent layers. This pattern suggests that the performance improvements of VIPAMIN-
may be largely attributed preventing collapse at the initial block. However, we posit that deeper orthogonality still plays a crucial role. Through the lens of representation oversmoothing deep transformers, prior work [12 [3] shows that self- attention layers reduce representational rank and act as low-par filters; our orthogonalization module explicitly increases the rank of self-attention outputs, counteracting this depth-wise tendency. Thus, although Figure S4 illustrates only early-layer prompt-side collapse, deep orthogonalization likely also mitigates oversmoothing of the attention pathway-a complementary failure mode consistent with the prior theory.
I.6 Supervised VPT vs. Self-Supervised VPT
Our study primarily investigates visual prompt initialization for VPT in self-supervised backbones. A natural extension is to ask whether similar failure modes-such uniform prompt attention-also arise in supervised VPT. Figure presents a comparison of prompt attention entropy across the 19 VTAB- 1k tasks (ordered by LP Ratio), contrasting VPT with self-supervised (MoCo-v3) and
35

---

Prompt Attention Entropy
4.8
4.6
Entropy
MoCo VPT
Supervised
Dissimilar
Similar
Sorted VTAB Datasets (LP/FT)
Figure S5: Comparison prompt attention entropy ( across VTAB-Ik tasks for VPT using MoCo- v3 (self-supervisee) and supervised pretrained backbones.
74
73
Accuracy (%)
72
71
70
69
SPT/rand
VIPAMIN
Figure S6: We conduct 10 independent training with the best performing hyperparameter and report the distribution of test accuracy of MAE pretrained model on Resisc45 dataset.
supervised backbones. The results show that VPT with a supervised backbone consistently exhibits lower entropy, indicating a greater tendency to focus attention on smaller subset of tokens.
We attribute this behavior the nature of semantic representations learned through supervised pretraining. Prior work has shown that supervised ViT's tend to encode more class-aligned and semantically structured features the image level [60]. We hypothesize that, in this setting, even randomly initialized prompts can more easily attend to meaningful regions, thereby reducing the benefits of specialized prompt initialization techniques.
I.7 Stability Study
Prompt tuning is known to exhibit instability across random seeds, raising concerns about its repro- ducibility [6]. One key source of variability the order in which training data presented during optimization. To evaluate robustness to this factor, we conduct 10 independent runs of each method with different data orderings while keeping all other settings fixed, including the initialization point. As shown in Figure S6 VIPAMIN not only achieves the highest average accuracy but also exhibits significantly lower variance, indicating improved stability across random seeds.
36


Reviews:
[{"review_id": "BPCRpQmZyf", "reviewer": "Reviewer_1xA8", "summary": "This paper introduces a visual prompt initialization strategy, termed VIPAMIN, for self-supervised Vision Transformers that aims to tackle near-uniform prompt attention and collapse into the frozen embedding subspace of the VPT. VIPAMIN aligns each prompt with semantically coherent image tokens and injects an orthogonal component that expands representation space in single forward pass. Empirical evaluations on VTAB-1k and FGVC demonstrate improvement over VPT on few-shot classification tasks.", "questions": "Refer to weakness", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. This paper has solid motivation with both empirical and theoretical analysis on why VPT failed under distribution shift, attended every patch equally, and contributed little to diverse representational directions. VIPAMIN is proposed to mitigate the gap based on these observations.\n2. The proposed method relies on two matrix operation, which is comparatively efficient. \n3. VIPAMIN shows consistent better performance in few-shot benchmarks against VPT.\n\nWeakness:\n1. Currently experiments are exclusively on image classification, and the author should also consider few-shot settings in other tasks like segmentation and object detection to demonstrate this method's generalization beyond classification problems.\n2. VIPAMIN falls short on Specialized group, which requires novel semantic representations, even with the orthogonalizing module comparing to SPT. The author should include more discussion to this performance gap.\n3. The author claimed that vanilla VPT failed to attend to meaningful token. The author should compare with, or at least discuss, semantic driven VPT (e.g. DA-VPT[1]). \n4. The orthogonalizing module creates $p^{orth}$ by projecting a random vector prompt outside the frozen subspace. Although this expands rank, there's nothing ensures the new direction is meaningful for the task. \n\n[1] DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers, CVPR 2025", "was_revised": true, "final_justification": "The author has addressed my concerns during rebuttal.", "rebuttal": "We are grateful for the reviewer’s constructive input. We have taken the comments seriously and hope our responses adequately resolve the questions raised.\n\n---\n\n# W1: Limited task scope beyond image classification\n\nWe agree that evaluating beyond image classification would strengthen the generality claim. To examine the applicability of our method beyond image classification, we turned to the well-established SuperGLUE natural language benchmark, which spans diverse tasks such as coreference resolution, question answering, and natural language inference. Applied to **BERT‑Large** on **SuperGLUE**, VIPAMIN raises the mean accuracy by **+1%p** over P‑Tuning v2—a leading prompt method originally developed for the language domain—and surpasses all competing vision‑prompt baselines as well as standard prompt‑tuning variants (Table R1). These gains, achieved in a domain that differs substantially from vision, show that the method’s core mechanism—coupling task tokens with prompt tokens via matching and orthogonalization—generalizes beyond image classification.\n\n**Table R1**: Per-task results for SuperGLUE development set with a pretrained BERT-Large model.\n|  | BoolQ | CB | COPA | RTE | WiC | WSC | Mean |\n|---|---|---|---|---|---|---|---|\n| Full | 77.7 | 94.6 | 69.0 | 70.4 | 74.9 | 68.3 | 75.8 |\n| Prompt Tuning | 67.2 | 80.4 | 55.0 | 53.5 | 63.0 | 64.4 | 63.9 |\n| Prefix Tuning | 75.8 | 94.6 | 73.0 | 78.3 | 75.1 | 68.3 | 77.5 |\n| E^2VPT | 74.4 | 80.4 | 77.0 | 78.7 | 74.3 | 67.3 | 75.4 |\n| VFPT | 74.8 | 81.2 | 78.1 | 77.2 | 75.3 | 68.4 | 75.8 |\n| VIPAMIN-deep | 74.6 | 94.6 | 79.0 | 79.1 | 74.3 | 69.2 | 78.5 |\n\n---\n\n# W2: Performance gap on specialized tasks despite orthogonalizing module\n\nWe appreciate the reviewer’s observation. First, we emphasize that VIPAMIN achieves state-of-the-art performance among shallow prompt tuning methods across all VTAB-1k groups using MoCo pretraining, and falls short of SPT only in the Specialized group under MAE pretraining (see Table 1). To clarify this, we provide a per-task breakdown in Table R2, which shows that the performance gap is concentrated in medical datasets (Diabetic Retinopathy and Patch Camelyon), where success depends heavily on injecting domain-specific knowledge.\nWhile this may appear to reflect a failure of the orthogonalizing module, we argue otherwise for two reasons.\n\n1. As shown in Table 4, adding the orthogonalizing module yields the largest performance gain on the Specialized group (**+1.3%p**), indicating that the module does contribute positively.\n2. In our response to Weakness 4, we demonstrate that the orthogonalized bias learns directions that are more aligned with the downstream task in medical domains—supporting that the injected directions encourage the model to seek more specialization.\n\nWe also believe that SPT’s relative strength on these two datasets under MAE is **tied to task and pretraining bias**. Both Patch Camelyon and Diabetic Retinopathy are fundamentally aimed at detecting localized abnormal regions. MAE is known to preserve fine-grained local information better than MoCo [1], and SPT’s clustering-based prompt assignment likely benefits from this, as it aggregates local-token clusters. However, SPT’s mechanism does not **infer new semantic knowledge**, which is evident from its underperformance in the Structured group compared to VIPAMIN (**+4.0%p** on MAE). For example, in count-based reasoning tasks like CLEVR/count—where pretrained objectives offer little guidance—VIPAMIN’s semantic token selection and orthogonal bias injection lead to significant gains over SPT. Lastly, we note that VIPAMIN outperforms SPT/rand, the more feasible variant of SPT, on every task within the Specialized group, further reinforcing the robustness of our approach (Table R2).\nTaken together, we view the observed performance gap as a challenge of knowledge transfer in highly specialized domains, rather than a failure of our method. Nevertheless, we appreciate the point and will expand our discussion accordingly.\n\n**Table R2**:  Per task accuracy comparison on VTAB-1k, specialized group on MAE.\n|  | Patch Camelyon | EuroSAT | Resisc45 | Diabetic Retinopathy |\n|---|---|---|---|---|\n| SPT | 84.4 | 93.0 | 70.8 | 75.4 |\n| SPT/rand | 77.4 | 91.9 | 73.0 | 73.3 |\n| VIPAMIN | 80.1 | 92.8 | 73.3 | 73.7 |\n\n---\n\n# W3: Comparison with semantic-driven VPT methods\n\nWe agree that recent variants of VPT have begun to incorporate semantic information, and we now include a discussion of two representative methods: DA-VPT and iVPT. Ultimately, these approaches—and ours—differ in how they couple the image representation with the prompt. DA-VPT selects prompt membership based on **the CLS token of same-class images**, optimizing the assignments via metric learning. iVPT uses **cross-attention between CLS and image tokens** to identify salient regions, and then injects additive prompts into those regions to enhance representation.\nIn contrast, our matching module in VIPAMIN targets the **direct interaction between prompt and image tokens**, without relying on CLS-centric attention patterns. Our empirical analysis reveals a pathological attention collapse between prompt and image tokens in vanilla VPT, which our method explicitly mitigates. Thus, while all methods seek to improve semantic alignment, only our approach addresses the issue from a distinct perspective grounded in patch–prompt token dynamics. We compare VIPAMIN with other semantic-guided prompt methods discussed above (iVPT, DA-VPT) on the VTAB-1k benchmark. As shown in Table R3, VIPAMIN consistently outperforms these methods across all three task groups, suggesting that its design—particularly the way it couples patch and prompt tokens—provides **more effective guidance** for downstream adaptation.\n\n**Table R3**: Group test accuracy of VTAB-1k compared with semantic prompt methods\n|  | Additional Params | Natural | Specialized | Structured | Mean |\n|---|---|---|---|---|---|\n| DA-VPT | 1.6M | 74.2 | 83.2 | 55.2 | 68.1 |\n| iVPT | <0.1M | 76.1 | 84.5 | 57.9 | 70.2 |\n| VIPAMIN-deep | 0 | 77.7 | 84.8 | 58.8 | 71.2 |\n\n---\n\n# W4: Justification for task-relevance of orthogonalized direction\n\nFigure 2 analyzes interactions between prompt and image tokens, but whether the orthogonalizing module leads the model to acquire a **task‑pertinent** direction that actually improves classification remained open. We therefore evaluated the **last‑layer CLS representations (LLCR)** entering the final linear head. We treat the LLCR from a fully fine‑tuned model as an oracle, denoted $\\mathrm{LLCR}\\_{\\text{ft}}$. We then compare it with $\\mathrm{LLCR}\\_{\\text{no-orth}}$ (matching only) and $\\mathrm{LLCR}\\_{\\text{vipamin}}$ (matching + orthogonalizing). Using paired cosine distance per validation sample with a Wilcoxon signed‑rank test ($d\\_{\\mathrm{cos}} (\\mathrm{LLCR}\\_{\\text{ft}}, \\mathrm{LLCR}\\_{\\text{no-orth}}) - d_{\\mathrm{cos}} (\\mathrm{LLCR}\\_{\\text{ft}}, \\mathrm{LLCR}\\_{\\text{vipamin}})$), we found out that $\\mathrm{LLCR}\\_{\\text{vipamin}}$ is much closer to $\\mathrm{LLCR}\\_{\\text{ft}}$ than $\\mathrm{LLCR}\\_{\\text{no-orth}}$ in a statistical sense:\n\n- Patch Camelyon: p-value $< 10^{-3}$\n- Diabetic Retinopathy: p-value $< 10^{-6}$\n\nWe also measured the Grassmannian distance between the spans of the corresponding LLCR sets:\n\n- Patch Camelyon: $d_\\text{grass} (\\mathrm{LLCR}\\_\\text{ft}, \\mathrm{LLCR}\\_\\text{no-orth}) = 14.14$ vs. $d_\\text{grass} (\\mathrm{LLCR}\\_\\text{ft}, \\mathrm{LLCR}\\_\\text{vipamin}) = 13.94$\n- Diabetic Retinopathy: $14.56$ vs. $14.03$, respectively.\n\nLastly, we observe that greater closeness to the fine-tuned model consistently correlates with higher test accuracy.\n\n- Patch Camelyon: $\\text{Acc}\\_{\\text{no-orth}} = 82.22$ vs. $\\text{Acc}_{\\text{vipamin}} = 83.09$\n- Diabetic Retinopathy: $71.22$ vs. $74.52$, respectively.\n\nBoth tests indicate that the orthogonalized bias **steers representations toward the task‑adapted oracle**, supporting that the role of the orthogonalizing module is not merely rank‑inflating. As a final remark, we acknowledge that the injected direction is task‑agnostic at initialization—a consequence of our lightweight design. Nevertheless, this initial nudge consistently converges to task‑pertinent directions during training. Going forward, we plan to **regularize the orthogonal bias during training to reflect task characteristics more directly**, which we expect will further close the gap on highly specialized domains. \n\n---\n\n[1] Park et al., What do Self-Supervised Vision Transformers Learn?, ICLR 2023."}, {"review_id": "R5ISUbvu6j", "reviewer": "Reviewer_Cs3J", "summary": "The paper introduces VIPAMIN, a visual prompt initialization strategy addressing uniform attention and subspace collapse in self-supervised Visual Prompt Tuning (VPT).", "questions": "1. VIPAMIN does not surpass state-of-the-art methods (e.g., VFPT achieves 71.33% vs. VIPAMIN-Deep’s 71.23% on VTAB-1k MoCo-v3), weakening claims of superiority.\n2. Experiments omit validation under distribution shifts (e.g., corrupted data or domain gaps), despite identifying this as a core VPT failure mode.\n3. The orthogonalizing module lacks theoretical motivation (e.g., no formal proof of subspace collapse mitigation), and comparisons to SPT/rand overlook deeper methodological distinctions.\n4. Critical parameters k \\lamda are tuned empirically without analysis of their impact on stability or generalization.\n5. Figure 7 shows subspace collapse only in the first transformer block, yet VIPAMIN-Deep applies orthogonalization to all blocks without justification. This inflates computation without empirical benefit.", "limitations": "1. The method is tested only on ViT-B/16 backbones. Results may not hold for larger models or non-ViT architectures, limiting generalizability claims.\n2. The paper states VIPAMIN is \"lightweight\" but provides no latency/memory measurements versus baselines. Real-world overhead for orthogonalization (Eq. 9) and large-batch processing (B=256) remains unquantified.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Its strengths include a simple design requiring minimal computation and competitive results on VTAB-1k and few-shot benchmarks. However, significant weaknesses exist. The method fails to consistently outperform top baselines like VFPT in key experiments, lacks robustness validation under distribution shifts, and its novelty is limited by superficial comparisons to prior work. The orthogonalization mechanism lacks theoretical grounding, and critical implementation details (e.g., hyperparameter sensitivity) are underexplored.", "was_revised": true, "final_justification": "After consideration, I decide to keep my orginal score.", "rebuttal": "Thank you for the thoughtful review. We’ve done our best to respond to each concern as concretely as possible, and we hope our responses are helpful.\n\n---\n\n# Q1+L2: Questionable superiority of VIPAMIN-deep, and the lack of latency or memory comparisons\n\nWe acknowledge that VIPAMIN‑Deep underperforms VFPT by 0.10 %p in one isolated scenario, yet this is the **only** case across all our evaluations where we do not set the state of the art, in terms of mean accuracy. In contrast, our shallow–prompt results (Table 1) establish new SOTA on both MAE and MoCo backbones, exceeding the previous best by **+1.5%** each, and on FGVC few‑shot tasks (Table 2), VIPAMIN consistently surpasses prior methods at every shot count. \nFurthermore, as shown in Table R1, our method is **significantly more efficient** than other deep prompt-based approaches. We note that a more empirical breakdown is provided in Supplementary S1, which reports wall-clock runtimes for each component in VIPAMIN. Specifically, with a batch size of 256:\n\n- Data loading: ≈ 8 seconds (I/O‑dependent)\n- Matching module: ≈ 4 seconds\n- Orthogonalization module: ≈ 9 seconds\n\nThus, the total extra runtime over vanilla VPT is **< 30 s**. Memory overhead is also modest—max memory allocated is 2.2 GB on NVIDIA A6000. Given that these **one-time** costs can be amortized over the full training process, their practical overhead is near-zero, reinforcing that VIPAMIN is lightweight.\n\n**Table R1**: Computational overhead analysis on Flower-102\n|  | Tuned/Total | Additional FLOPs (over VPT) | Initialization Overhead (GFLOPs) | Note | Accuracy |\n|---|---|---|---|---|---|\n| VPT | 0.30% | 0 (17.7 GFLOPs) | 0 |  | 90.2 |\n| E2VPT | 0.20% | < 0 | 0 | Requires training twice | 91.3 |\n| iVPT | 0.31% | < 1M | 0 |  | N/A |\n| VFPT | 0.30% | 5.5M | 0 |  | 92.4 |\n| SPT | 0.30% | 0 | 3300 |  | 93.2 |\n| VIPAMIN-deep | 0.30% | 0 | 1.73 |  | 94.0 |\n\n---\n\n# Q2: Validation under distribution shifts\n\nWe agree that robustness under distribution shift is an important manifestation of the VPT failure mode. To address the question, we ran additional OOD tests on CIFAR-100 by corrupting the test set with multiple shifts. Table R2 shows that our method attains the best accuracy on 5 out of 6 corruptions and the highest mean accuracy over baselines (**+4.0%p** over SPT/rand). We will add these results and clarify that **distribution shift robustness is a consequence of fixing VPT’s adaptation failure**, not a separate objective.\n\n**Table R2**: OOD generalization test on CIFAR-100\n|  | Brightness | Contrast | Rotation | Gaussian | Shot | Speckle | Mean |\n|---|---|---|---|---|---|---|---:|\n| VPT | 36.2 | 20.9 | 6.8 | 13.2 | 21.4 | 25.5 | 20.6 |\n| SPT/Rand | 65.5 | 47.2 | 16.8 | 20.2 | 37.0 | 44.5 | 38.5 |\n| VIPAMIN | 71.4 | 57.5 | 18.7 | 19.8 | 39.9 | 48.8 | 42.5 |\n\n---\n\n# Q3-1: Theoretical grounding on orthogonalizing module\n\nWhile we do provide **the theoretical motivation** for our design, we acknowledge that a clearer theoretical justification on **how it mitigates subspace collapse** would help. In our preliminaries (line 112-119), we derived that the prompted self-attention output decomposes into a linear combination of the no-prompt self-attention output, $SA(X_0)$, and the prompt-induced component $P_0 W_V$. Therefore, we claimed that the linear bias injected by the prompts is important for shifting representations towards task-relevant semantics.\n\nHowever, even on highly dissimilar tasks, the prompt collapses into the original self-attention subspace (Fig. 2(b)). Formally, let\n$A = P_0 W_V \\in \\mathbb{R}^{n_p \\times d}$,\n$B = S A(X_0) \\in \\mathbb{R}^{n_e \\times d}$,\nwith row-ranks $r_A, r_B$ (We assume $n_p=n_e=n$ for now). Without orthogonalization, collapse implies $\\mathrm{row}(A) \\subseteq \\mathrm{row}(B)$, hence\n\n$dim(\\mathrm{row}(A)+\\mathrm{row}(B))=dim(\\mathrm{row}(B))=r_B,$\n\nso the prompt fails to add new directions. \nOur orthogonalizing module chooses $P^{\\text{orth}}$ so that $\\mathrm{row}(A') \\subseteq \\mathrm{row}(B)^\\perp$, yielding\n\n$dim(\\mathrm{row}(A')+\\mathrm{row}(B))=\\min(d,r_{A'}+r_B),$\nthereby strictly increasing the available subspace whenever $r_{A'} > 0$. The prompt mitigates collapse by adding directions outside the frozen subspace spanned by $B$. For completeness, when the attention score matrix $S \\in \\mathbb{R}^{n_e \\times n_p}$ is applied to $A$, $\\mathrm{row}(S A) \\subseteq \\mathrm{row}(A)$; thus the above row-space relations still hold. In short, **the orthogonalizing module guarantees additive subspace capacity beyond the frozen self-attention span**, which aligns with the observed gains. \n\n\n# Q3-2: Clarifying methodological differences with SPT/rand\n\nConcerning **SPT/rand**, it initializes prompts by randomly selecting embedding tokens, assuming that mirroring embeddings aids adaptation. In contrast, our method introduces a specialization mechanism via hyperparameter $k$, enabling each prompt to focus on a targeted subset of embeddings for task-adaptive attention.\nFrom a technical standpoint, VIPAMIN’s matching module and SPT/rand share a superficial similarity—both introduce randomness when coupling embedding tokens to prompts. SPT/rand, however, **uniformly** selects $n_p$ tokens at random from the entire training set of $|D_\\text{train}| \\times n_e$. In contrast, VIPAMIN first exploits the natural coherence of key‑space representations (Fig. 3), then uses random vectors to select clusters and assigns $k$ embedding tokens from each selected cluster to the prompt. This strategy makes each prompt an **expert for a semantically coherent region**, while randomness ensures that different prompts acquire complementary expertise.\n\n---\n\n# Q4: Justification for critical hyperparameter choices—on stability and generalization\n\nWhile VIPAMIN is fairly stable for $k$ and $\\lambda$, they still do bear a significant impact on generalization. We measured the coefficient of variation (CV) of validation accuracy across a grid of $k$ and $\\lambda$ values with fixed learning rate. **Across all VTAB-1k datasets and learning rates**, CV remained below 0.1 with only a few exceptions–SmallNORB and Clevr datasets–where it was still modest(<0.2). These results support that the method is reasonably robust to hyperparameter variation. Furthermore, our paper provides a **detailed analysis of the roles of hyperparameters** in Fig. 4(c). The figure shows that for tasks with large distribution shifts, smaller $k$ and larger $\\lambda$ work best—vice versa for relatively in-domain tasks. Building on this insight, we aim to explore automated hyperparameter selection in future work, conditioned on task difficulty or domain shift severity.\n\n---\n\n# Q5: Mismatch between Figure 7 and depth-wide orthogonalization on VIPAMIN-deep\n\nWhile we agree that Fig. 7 suggests that deep orthogonalization may be redundant, our empirical findings show otherwise. Applying orthogonalization only to the first block leads to degraded performance (84.5% $\\to$ 78.7%), indicating that addressing early collapse alone is insufficient.\nWe interpret this through the lens of **representation oversmoothing** in deep transformers. Prior studies [1,2] have shown that self-attention layers tend to reduce representational rank, behaving as low-pass filters. This rank collapse has been linked to impaired model scalability. Notably, our orthogonalization module **explicitly increases the rank** of self-attention outputs, counteracting the inherent rank collapse of the transformer.\nTherefore, even though Fig. 7 focuses on early-layer **prompt-side** collapse, we believe that deep orthogonalization mitigates depth-wise **oversmoothing of the attention output itself**—a distinct yet complementary failure mode. This effect is not directly visualized in the figure, but it is supported by the performance increment (+5.8%p) and aligns with prior theoretical insights.\n\n---\n\n# L1: Addressing generalizability over ViT-Base\n\nWhile our main tables focus on ViT-B/16, we included Figure 4(b) in the main paper to provide initial evidence that our method also extends to ViT-Large and Huge. That said, we acknowledge that our initial results were limited in three aspects: **modality, model architecture, and scale**. To address this, we conducted additional experiments on the **HiViT**[3], a hierarchical vision model, and observed consistent improvements over baselines (Table R3). We also applied our method to **BERT-Large** on the **SuperGLUE** benchmark to test generality across modality and scale. As shown in Table R4, our method outperforms both **P-Tuning v2** and **full fine-tuning**, two of the strongest baselines in the language domain. These results strengthen our generalization claim across architecture, modality, and scale.\n\n**Table R3**: Group test accuracy on HiViT for VTAB-1k\n| | Natural | Specialized | Structured | Mean |\n|:---:|:---:|:---:|:---:|:---:|\n| VPT | 62.5 | 78.6 | 53.2 | 67.9 |\n| SPT/rand | 63.7 | 79.3 | 54.5 | 68.7 |\n| VIPAMIN | 64.0 | 79.7 | 58.0 | 69.0 |\n\n**Table R4**: Per-task results for SuperGLUE development set\n\n| | BoolQ | CB | COPA | RTE | WiC | WSC | Mean |\n|---|---|---|---|---|---|---|---|\n| Full | 77.7 | 94.6 | 69.0 | 70.4 | 74.9 | 68.3 | 75.8 |\n| Prompt Tuning | 67.2 | 80.4 | 55.0 | 53.5 | 63.0 | 64.4 | 63.9 |\n| P-Tuning v2 | 75.8 | 94.6 | 73.0 | 78.3 | 75.1 | 68.3 | 77.5 |\n| E2VPT | 74.4 | 80.4 | 77.0 | 78.7 | 74.3 | 67.3 | 75.4 |\n| VFPT | 74.8 | 81.2 | 78.1 | 77.2 | 75.3 | 68.4 | 75.8 |\n| VIPAMIN-deep | 74.6 | 94.6 | 79.0 | 79.1 | 74.3 | 69.2 | 78.5 |\n\n---\n\n[1] Dong et al., Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ICML 2021.\n\n[2] Wang et al., Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice, ICLR 2022.\n\n[3] Zhang et al., Hivit: A simpler and more efficient design of hierarchical vision transformer, ICLR 2023."}, {"review_id": "C86ksugl4Z", "reviewer": "Reviewer_MNuV", "summary": "Identifying the uniform attention and subspace collapse limitation, this paper introduces VIPAMIN, a initialization technique that addresses these issues via attention-guided matching and orthogonal subspace injection. Specifically, VIPAMIN aligns prompts with semantically informative regions in the embedding space, and injects representational directions beyond the pretrained subspace. VIPAMIN are validated under few-shot settings.", "questions": "1. In Figure 1, the importance of “semantic token selection” and “bias injection,” are observed, but is there a more explicit visualization? For example, which specific semantic tokens are selected, and what precise biases are being injected?\n2. In Figure 2, the “Failure modes of VPT” are mentioned. To which modes does “mode” refer? Please clarify the specific failure modes being illustrated.\n3. In Table 3, what does the metric labeled “Spec” denote? Its performance is inferior to several baselines—can the authors provide a detailed analysis explaining this discrepancy?", "limitations": "yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper introduces several novel technical modules—specifically, “Selecting Tokens for Prompt Propagation” and “Injecting Semantic Bias into Attention Outputs”—both of which represent innovative contributions.\n\n**Weaknesses**\n\n1. **Lack of Detailed Visualization for Semantic Token Selection and Bias Injection**\n   Although Figure 1 suggests the importance of “semantic token selection” and “bias injection,” the paper does not provide explicit visualizations that identify which tokens are being selected or which biases are being injected. Without such detailed illustrations, it is difficult to evaluate whether the selected tokens indeed capture the intended semantics or whether the injected biases align with the theoretical motivation.\n\n2. **Unclear Definition of “Failure Modes” in VPT**\n   The term “mode” in Figure 2’s “Failure modes of VPT” is not defined, leaving the reader uncertain about the specific failure cases being depicted. This ambiguity prevents a clear understanding of the limitations of VPT and hinders the ability to assess whether the proposed method effectively addresses those failure modes.\n\n3. **Ambiguity Surrounding the “Spec” Metric in Table 3**\n   Table 3 reports a metric labeled “Spec,” but the paper does not define this term. Furthermore, “Spec” underperforms relative to baseline methods, yet no detailed analysis is provided to explain why. The absence of a clear definition and an in-depth investigation into the poorer performance undermines the reader’s ability to interpret the results and understand the method’s limitations.", "was_revised": true, "final_justification": "Thank you! I keep my current rating.", "rebuttal": "We appreciate the reviewer’s feedback and have reflected on each point carefully. We hope our responses make the key aspects of our method clearer.\n\n---\n\n# Q1: More explicit illustration of semantic token selection and bias injection\n\nWe appreciate the interest in more explicit visualizations. Regarding semantic token selection, the primary goal of matching module is to **identify token groups that share consistent semantic meaning**. Figure 3 provides such an illustration: the output of the matching module, denoted as red vectors and visible patches in the image, shows the module is effectively identifying semantically coherent visual elements—e.g., one group focuses on a shelf displaying soft bread, while another attends to a separate shelf with crusty bread. This coherence confirms that our matching module isolates semantically aligned token sets across instances.\n\nAs for orthogonal bias injection, this mechanism introduces directions orthogonal to task-specific features, encouraging the model to **explore novel subspaces**. Since these directions are not tied to localized visual regions, they are inherently less suitable for image-based visualization as in the former case. However, our analysis on the last layer representation of CLS tokens demonstrates that these injected directions are non-trivial and desirable for improving adaptability.\n\nWe analyzed **the last-layer CLS representation (LLCR)** since it directly influences the model’s final prediction. To test the impact of orthogonal bias injection, we also trained a variant using only the semantic token selection (no-orth). As a reference, we treated the fully fine-tuned model (ft) as an oracle that captures novel patterns beyond what was learned during pretraining. We compare $\\mathrm{LLCR}\\_{\\text{ft}}$ with $\\mathrm{LLCR}\\_{\\text{no-orth}}$ (matching only) and $\\mathrm{LLCR}\\_{\\text{vipamin}}$ (matching + orthogonalizing). Using paired cosine distance per validation sample with a Wilcoxon signed‑rank test ($d_{\\mathrm{cos}} (\\mathrm{LLCR}\\_{\\text{ft}}, \\mathrm{LLCR}\\_{\\text{no-orth}}) - d_{\\mathrm{cos}} (\\mathrm{LLCR}\\_{\\text{ft}}, \\mathrm{LLCR}\\_{\\text{vipamin}})$), we found out that $\\mathrm{LLCR}\\_{\\text{vipamin}}$ is much closer to $\\mathrm{LLCR}\\_{\\text{ft}}$ than $\\mathrm{LLCR}\\_{\\text{no-orth}}$ in a statistical sense:\n\n- Patch Camelyon: p-value < $10^{-3}$\n- Diabetic Retinopathy: p-value < $10^{-6}$\n\nWe also measured the Grassmannian distance between the spans of the corresponding LLCR sets:\n\n- Patch Camelyon: $d_\\text{grass} (\\mathrm{LLCR}\\_\\text{ft}, \\mathrm{LLCR}\\_\\text{no-orth}) = 14.14$ vs. $d_\\text{grass} (\\mathrm{LLCR}\\_\\text{ft}, \\mathrm{LLCR}\\_\\text{vipamin}) = 13.94$\n- Diabetic Retinopathy: $14.56$ vs. $14.03$ respectively.\n\nLastly, we observe that greater closeness to the fine-tuned model consistently correlates with higher test accuracy.\n\n- Patch Camelyon: $\\text{Acc}\\_{\\text{no-orth}} = 82.22$ vs. $\\text{Acc}\\_{\\text{vipamin}} = 83.09$\n- Diabetic Retinopathy : $71.22$ vs. $74.52$, respectively.\n\nBoth tests indicate that **the orthogonalized bias steers representations toward the task‑adapted oracle**.\nIn summary, semantic token selection (via the matching module) and orthogonal bias injection (via the orthogonalizing module) work in tandem to shape more desirable representations, consistently leading to performance gains. The quantitative contributions of each module (Table 4) and the Grad-CAM visualizations (Figure 5) further support their roles in learning more effective and discriminative features.\n\n---\n\n# Q2: Unclear specification of failure modes illustrated in Figure 2\n\nThe left plot in Figure 2 illustrates the failure mode of **uniform attention, where prompts in VPT fail to select the relevant image patches**, while the right plot highlights VPT’s inability to **acquire novel semantic biases, exemplified by the collapse of prompt space into the self attention output space of embedding tokens**. These two failure modes—uniform attention and the lack of semantic bias acquisition—**interact to produce the consistent underperformance of VPT** on tasks that are substantially divergent from the pretraining domain, as illustrated in Figure 1(b). Furthermore, we elaborate on this pattern in Figure A1, where we show that such underperformance on dissimilar tasks is a general trend across multiple self-supervised VPT variants (e.g., MoCo, MAE, DINO, MAE-Tiny).\n\n---\n\n# Q3: Unclear meaning and underperformance of “spec”\n\nThroughout the paper, we used “spec” as an abbreviation for Specialized groups in VTAB-1k, including EuroSAT, Resisc45, Patch Camelyon, and Diabetic Retinopathy datasets. We are sorry for the ambiguity of the term. We will clarify that “Spec” is used as a shorthand for “Specialized” in the camera-ready version. Regarding the relative underperformance in the specialized domain, we break down into two scenarios—**Shallow-based prompt in MAE model** (Table 1), **Deep-based prompt** (Table 3).\nFirst, in the shallow-based prompt in the MAE model, we were able to see that the performance gap is confined to two medical datasets (Patch Camelyon, Diabetic Retinopathy) under MAE pretraining (Table R1). These medical tasks aim to detect fine‑grained local anomalies. MAE is known to preserve fine-grained local information better than MoCo [1], and SPT’s clustering-based prompt assignment likely benefits from this, as it aggregates local-token clusters. However, SPT’s mechanism **does not infer new semantic knowledge**, which is evident from its underperformance in the Structured group compared to VIPAMIN. For example, in count-based reasoning tasks like CLEVR/count—where pretrained objectives offer little guidance—VIPAMIN’s semantic token selection and orthogonal bias injection lead to significant gains over SPT (**+4.0%p** on MAE). Lastly, we note that VIPAMIN outperforms SPT/rand, the more feasible variant of SPT, **on every task within the Specialized group**, further reinforcing the robustness of our approach (see Table R1). Taken together, these results indicate that the gap stems from the unique demands of highly specialized domains rather than from any fundamental limitation of VIPAMIN itself.\n\nFor the deep-based prompt model, despite an accuracy gap in the Specialized group, VIPAMIN‑Deep offers the highest accuracy‑per‑cost ratio. Table R2 shows it delivers the best **accuracy‑to‑cost trade‑off**. Competing deep prompt methods incur heavy overheads—e.g., E2VPT roughly doubles training wall‑clock time due to its prune‑and‑rewind cycle, and SPT can require days to generate its initialization. By contrast, VIPAMIN’s initialization is produced in under half a minute, even for larger models and datasets, making our approach far more efficient while remaining highly competitive in accuracy.\n\n\n**Table R1**:  Per task accuracy comparison on VTAB-1k, specialized group on MAE.\n\n|  | Patch Camelyon | EuroSAT | Resisc45 | Diabetic Retinopathy |\n|---|---|---|---|---|\n| SPT | 84.4 | 93.0 | 70.8 | 75.4 |\n| SPT/rand | 77.4 | 91.9 | 73.0 | 73.3 |\n| VIPAMIN | 80.1 | 92.8 | 73.3 | 73.7 |\n\n**Table R2**: Computational overhead analysis on Flower-102\n\n|  | Tuned/Total | Additional FLOPs (over VPT) | Initialization Overhead (GFLOPs) | Note | Accuracy |\n|---|---|---|---|---|---|\n| VPT | 0.30% | 0 (17.7 GFLOPs) | 0 |  | 90.2 |\n| E^2VPT | 0.20% | < 0 | 0 | Requires training twice | 91.3 |\n| iVPT | 0.31% | < 1M | 0 |  | N/A |\n| VFPT | 0.30% | 5.5M | 0 |  | 92.4 |\n| SPT | 0.30% | 0 | 3300 |  | 93.2 |\n| VIPAMIN-deep | 0.30% | 0 | 1.73 |  | 94.0 |\n\n---\n\n[1] Park et al., What do Self-Supervised Vision Transformers Learn?, ICLR 2023."}]

Author rebuttal:

Meta-review: This paper introduces VIPAMIN, a novel prompt initialization technique that improves the prompt-tuning-based specialization of self-supervised models to downstream tasks. The tackled problem is interesting and timely, and the proposed solution simple and effective. The reviewers have noted some weak points that have been tackled by the authors during the rebuttal phase. Overall, this is a good paper and I recommend acceptance as a Poster.
Decision: Accept (poster)