Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts
Authors: Minh Le
Anh Nguyen
Huy Nguyen
Chau Nguyen
Anh Tuan Tran
Nhat Ho
Venue: ICLR 2026 Poster (2026)
Keywords: mixture of experts
visual prompt tuning
theory
parameter-efficient fine-tuning
pre-trained model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Visual Prompt Tuning (VPT) has proven effective for parameter-efficient adaptation of pre-trained vision models to downstream tasks by inserting task-specific learnable prompt tokens. Despite its empirical success, a comprehensive theoretical understanding of VPT remains an active area of research. Building on the recently established connection between Mixture of Experts (MoE) and prompt-based methods, wherein each attention head can be conceptualized as a composition of multiple MoE models, we reinterpret VPT as the introduction of new *prompt experts* into these MoE structures. We identify a key limitation in existing VPT frameworks: the *restricted functional expressiveness* of prompt experts, which remain static and thus limited in their adaptability. To address this, we propose **Visual Adaptive Prompt Tuning (VAPT)**, a novel method that endows prompt experts with enhanced expressiveness while preserving parameter efficiency. Empirical evaluations on VTAB-1K and FGVC demonstrate that VAPT achieves *substantial performance improvements*, surpassing fully fine-tuned baselines by **7.34%** and **1.04%**, respectively. Moreover, VAPT consistently outperforms VPT while *requiring fewer additional parameters*. Furthermore, our theoretical analysis indicates that VAPT achieves optimal sample efficiency. Collectively, these results underscore the theoretical grounding and empirical advantages of our approach.

Published as a conference paper at ICLR 2026
REVISIT VISUAL PROMPT TUNING: THE EXPRESSIVE- NESS OF PROMPT EXPERTS
Minh St Anh Nguyen"), Huy Nguyen?, Chau Nguyen"!, Anh Tran', Nhat Ho
1 Qualcomm Al Research 2 The University of Texas at Austin
Trivita AI
Hanoi National University of Education
ABSTRACT
Visual Prompt Tuning (VPT) has proven effective for parameter-effiient adapta- tion of pre-trained vision models downstream tasks by inserting task-specific learnable prompt tokens. Despite its empirical success, comprehensive theoretical understanding of VPT remains an active area research. Building on the recently established connection between Mixture of Experts (MoE) and prompt-based meth- ods, wherein each attention head can be conceptualized as composition of multiple MoE models, reinterpret VPT as the introduction of new prompt experts into these MoE structures. We identify a key limitation in existing VPT frameworks: the restricted functional expressiveness of prompt experts, which remain static and thus limited in their adaptability. To address this, we propose Visual Adaptive Prompt Tuning (VAPT), novel method that endows prompt experts with enhanced expres- siveness while preserving parameter efficiency. Empirical evaluations on K and FGVC demonstrate that VAPT achieves substantial performance improvements, surpassing fully fine-tuned baselines by 7.34% % and 1.04% % respectively. Moreover, VAPT consistently outperforms VPT while requiring fewer additional parame- ters. Furthermore, our theoretical analysis indicates that VAPT achieves optimal sample efficiency. Collectively, these results underscore the theoretical ground- ing and empirical advantages of our approach. Our code is publicly available at https: / thub.. com htwp:i..coo//iihhhhiaiiia www..//inicccayetttccnnnn///aa
INTRODUCTION
Foundational vision models, pre-trained on large-scale datasets (Dosovitskiy) 2020 Radford et al. 2021 Kirillov et al., 2023), have demonstrated remarkable success and robust generalization across a wide range of computer vision tasks. As result, fine-tuning these models for specific downstream tasks has become a widely adopted paradigm (Iofinova et al. 2022). However, fully fine-tuning large foundational models can be computationally prohibitive, leading to growing interest in parameter- efficient fine-tuning (PEFT) techniques (Cai et al.) 2020 Zhang et al. 2020 Hu et al. 2021), which update only small subset of parameters. Among these methods, Visual Prompt Tuning (VPT) (Jia et al. 2022) has emerged as simple yet powerful approach that appends learnable prompt tokens to the input, which serve as task-specific instructions to guide the pre-trained transformer model. Despite VPT's empirical effectiivness, its theoretical underpinnings remain an active area of research (Petrov al.
2023 Oymak et al. 2023 Wang et 2024).
Recently, Le et al. (2024) established a formal connection between attention mechanisms (Vaswani 2017). prompt-based methods, and Mixture of Experts (MoE) models (Jacobs et al., 1991 Shazeer et al. 2017), yielding new insights into the design and optimization of prompting strategies. Their analysis demonstrates that each attention head transformer can equivalently interpreted as composition of nultiple MoE models stacked together. Within this framework, VPT corresponds to fine-tuning these implicit, pre-trained MoE structures by introducing new, learnable prompt experts. These prompt experts collaborate with the pre-trained experts facilitate effective task adaptation.
Equal contribution Work done while at Qualcomm
Qualcomm AI Research is initiative of Qualcomm Technologies, Inc.

---

Published as a conference paper at ICLR 2026
This connection opens avenues for deeper theoretical investigations and the development of advanced strategies for prompt-based learning (Le et al., 2025b at Diep et al. 2025).
Building on this MoE interpretation, identify key limitation in the standard formulation of VPT. While the pre-trained experts within attention heads are linear functions of the input features, the newly introduced prompt experts modeled constant, input-nnariint vectors. Given that effective adaptation relies on collaboration between these prompt and pre-trained experts, we hypothesize that this functional disparity, specifically, the restricted expressiveness of static prompts, may constrain VPT's adaptation efficacy. This design also deviates from standard MoΣ practices, where experts are typically designed be aaput-adaptive. However, increasing the expressiveness of prompt experts raises concerns about parameter and computational overhead. This leads us to the following question:
(Q) Can we improve model performance by increasing the expressiveness of prompt experts while maintaining parameter efficiency?
To answer this question, we propose Visual Adaptive Prompt Tuning (VAPT), novel approach that incorporates input-dependent prompt experts while maintaining the parameter efficiency characteristic of VPT. The VAPT design comprises two main components: token-wise projectors and a shared feature projector, which leverage global information from imput features to generate adaptive prompt tokens. A key advantage of VAPT is its efficiency: it achieves enhanced expressiveness minimal computational overhead, adding only 0.6 % FLOPs relative to VPT and requiring fewer trainable parameters. Additionally, the token-wise projectors and channel-wise convolutions result in structurally simple formulation of the prompt experts. This simplicity enables rigorous theoretical analysis (see Section 5). a contribution largely absent in the prior prompt-tuning literature. Our analysis demonstrates that VAPT attains optimal sample efficiency for prompt estimation. Empirical results strongly corroborate these theoretical findings. For example, in a low-data regime on the Stanford Dogs dataset (Khosla al., 2011) I) (using only 1% of training data), VAPT achieves 60.1 % accuracy, a stark contrast to VPT's s 3.6 %. Moreover, on standard benchmarks VTAB- IK (Zhai et al.) 2019) and FGVC QJa et al. 2022), VAPT significantly improves performance over fully fine-tuned baselines by 7.34% % and 1.04% %, respectively. Crucially, VAPT consistently outperforms VPT across benchmarks despite utilizing fewer parameters, underscoring its theoretical and empirical strengths.
Contributions. 1. From MoE perspective, we identify key limitation in the formulation of VPT: its prompt experts are input-invariant, thereby constraining their functional expressiveness. 2. We introduce VAPT, novel formulation that injects imput-adaptive prompt experts while preserving the parameter-efficiency of VPT. 3. VAPT's simple formulation enables theoretical analysis to validate its effectiveness, an aspect largely missing in prior work. Our theoretical analysis demonstrates that VAPT attains optimal sample efficiency for prompt estimation, providing a rigorous foundation for its practical value. 4. Extensive experiments show that VAPT consistently surpasses VPT with fewer trainable parameters, validating its effectiveness and efficiency both thcoretically and empirically.
Notation. For n E N, let [m] {1, ...,,... n). For a set S, |S| denotes its cardinality. Given vector w- = (w1, u2:..., ua) E Rd and a = (01,02:-
(t)) E N4, define v = un u² ...u. uu |u| U1 + u2 ... + Ue and a! aylay!... rog!...ac a.l.. Let [|u denote the Euclidean norm of u. For positive sequences (a,),)) nn>1 and (b,)m21, we write an O(b,) or an S b, if a, < Cb, for all n for some C > The notation an = (r(,,) indicates on (b, is stochastically bounded.
2 BACKGROUND
In this section, we first review Visual Prompt Tuning in Section 2.1 and then the Mixture of Experts model in Section 2.2 Additional related work is presented in Appendix C
2.1 VISUAL PROMPT TUNING
Vision Transformer (ViT) (Dosoviskkiy 2020) has proven be a powerful backbone architecture for visual recognition. A ViT contains L blocks, each comprising a multi-head self-attention (MSA) layer followed by feed-forward network (FFNN. For clarity, we consider the I-th ViT block. Let
X (1) = " (1)
E R Nxd be the input tokens, where N is the sequence length, and d is the (1) E Nxd queries,
embedding dimension. The MSA layer processes x X XV
2

---

Published as a conference paper at ICLR 2026
keys, and values, producing: MSA (X ()) (hn, hMS W E BR
hm = Attention(X x
m x'W") ER
(1)
for [M], where M is the number of attention heads, Wh Rdxd, w Raxde W Rdxd, and Wo RMd,,x xd are projection matrices with dg = de M
2022) adapts a pre-trained ViT by appending N2 learnable Visual Prompt Tuning (VPT) (Jia et al.
prompt parameters p())
(1)
PN, RR-, (where N, is the prompt length), to the
pi
input of each ViT block, thereby modifying the MSA layer's output as follows:
MSA(X (1) p()) Concat (.., hw)
X p())
Xx p())
X p())
hm
Attention
W
W
hm.1.-... ................................ Fmm.N+N,) ]"
E ER(N+N))x).
(2)
During training, only the prompts p(t) and the classification head are updated, while all ViT weights, including Wº. W.K WY and Wo. remain frozen. Within the VPT framework, MSA output tokens corresponding to the imput prompt locations are discarded and replaced by the prompts of the next layer 1((+1) before these tokens enter the subsequent block. Consequently, in Equation (2). hm,N+1 Nll........ Nm,N+ N, are not utilized downstream and their computation can therefore be bypassed.
2.2 MIXTURE OF EXPERTS
Mixture of Experts (MoE) is class of statistical machine learning frameworks that combines multiple models, known as experts, to produce more expressive and accurate predictions (Jacobs et al. 1991
Jordan & Jacobs 1994). An MoE model typically consists of N' expert functions, fi R R de for i e [N'), along with gating function, G IRd NN' which assigns weights experts based on
learned score functions, Rdd +
IR. For given input h E IRd, the MoE model generates output as:
y = Σ G(h); fz(h) = Σ exp (s)(())))
N
(h),
N' Ei=1 exp (st(h)
where Ghh) softmax(s1 (h) ..... (h)). Recent studies reveal connections between the atten- tion mechanism, prompt- based methods, and MoE frameworks (Le et al., 2024 2025b), presenting new promising opportunities investigate prompt-based techniques through MoE lens.
3 MOTIVATION
Mixture of Experts meets Visual Prompt Tuning. We begin by discussing the connection be- tween VPT and MoE, Following the notation from Section 2.1 and Equation (2). let (1)
E RNd be the concatenation of all N input tokens to the 1-th MSA layer.
11
IN
For notational simplicity, the layer superscript (I) is omitted the subsequent derivations. As shown in Le et al. (2024), each output vector hm, within the m-th attention head can be equivalently expressed as the output of an MoE model with input X. Specifically, define the set of experts as:
fx( W.V E, X W.V. Zj.
(3)
fN+ W-V T Py¹-
(4)
where j E [N] and j' E [Np]. The corresponding score functions are defined as:
(x)(X) X E WOWK E,X
x. WQWK xj
(5)
Vd,
Vd.
X E W WK
KT Py
x. TWWK P
81,++
(6)
d
Vdg
3

---

Published as a conference paper at ICLR 2026
for i [M]. Here, E, E R is two-dimensional matrix such that E,X- Zy. Finally, the output of the m-th attention head for the i-th token can be expressed as follows:
N exp(st,(X)) f((X) +
exp(s).43((X)) fN+ (X).
hm
(7)
j=1 N+N exp(sLu(X))
N-N exp($t,((X))
These formulations reveal that each attention head in ViT implicitly encodes multiple MoE, models, 1 for E [.V] This contrasts with conventional MoE layers (Shazeer et al. 2017). where experts and their gating functions typically operate on individual token embeddings z. In our formulation, the expert networks and their score functions process the entire input sequence X. Furthermore, these initial experts fi..... fN and score functions are inherently part of the pre-trained VIT, with their parameters embedded within the model weights (see Equations (3) and (,) thereby constituting pre- trained experts. Meanwhile, the new prompt experts fN-1.... JN+NN whose learnable parameters are contained within prompts (see Equations (4) and (6), are introduced to efficiently adapt the model downstream tasks. Consequently, VPT can be viewed an efficient method for fine-tuning
these implicit MoE. models by adding new prompt experts. These added experts effectively act as downstream task experts, enabling specialization for new tasks without retraining the entire model.
Restricted Functional Expressiveness of Prompt Experts. Equation (4) reveals a key limitation of prompt experts fl++....:; JN+ Although they are formally functions of X, they are represented by fixed prompt vectors ..,... -NN, that remain constant regardless of the input. While their associated score functions sy,N+,' Equation (6) are input-dependent linear functions, the functional form realized by the prompt expert itself is static. This departs from rypical MoE usage in the literature, where experts are adaptive functions of the input. Furthermore, contrasts with the pre-trained experts fI...., JN. which are linear functions of X (see Equation 3 ), making them comparatively more expressive. We hypothesize that this limited flexibility may constrain the effectiveness of VPT as fine-tuning strategy. Supporting this, prior work Petrov et al. (2023) demonstrates that prompt tuning can only add a bias term to the output of an attention block, thereby restricting representational capacity. This observation naturally motivates the central question: Can the performance of visual prompt tuning be improved by making prompt experts more expressive? Addressing this, we propose novel prompt formulation to enhance their expressiveness in Section
Balancing Expressivity and Efficiency. Despite the limitations noted above, one of the main advantages of the current prompt design in VPT is its simplicity. As indicated by Equation 4
each prompt expert requires only d parameters, making it highly parameter-efficient. One might consider naive linear design for a prompt expert fN43' -g.d + IRd., where fN+j/ (X) = W X. However, this approach would necessitate up N Ndxd d, parameters, drastically increasing storage and computational overhead compared to the current approach, especially given the high dimensionality Nd of the input X. This illustrates the core design challenge: enhancing prompt expert expressiveness without sacrificing the parameter efficiency that makes VPT attractive. Our objective is therefore to devise a prompt mechanism that effectively balances these competing requirements.
4 VAPT: VISUAL ADAPTIVE PROMPT TUNING
To investigate adaptive prompting, this section presents our proposed method, VAPT, which integrates two key modules: token- wise projectors and feature projector. These components are detailed in Section 4.1 and Section 4.2 respectively. Figure [ illluttrates the overall VAPT architecture.
4.1 AGGREGATING GLOBAL INFORMATION
To rigorously design the prompt experts, we first re-examine the formulation of the input and pre- formed by concatenating N feature tokens. These tokens can be spatially organized into the feature map Ximg E "f xd where H and W are its height and width, respectively, and N = И x W. Given that each token x, E Rdd corresponds to small patch of this feature map, Equation 33) implies that each pre-trained expert f; process their respective patches x; independently. Consequently. these experts are inherently limited to capturing only local information within the feature map.
trained experts. The input is defined as X = [xT ...,,,...;) xN]
Token-wise Projectors. As prompt experts are designed to collaborate with pre-trained experts for adaptation to downstream tasks, it is desirable for them specialize in information complementary
4

---

Published as a conference paper at ICLR 2026
Image tokens
Block 1-th
Layer Norm
Channel-wise Convolution
ViT block
X
VAPT
Token-wise Projectors
p())
Globally shared Feature Projector
Prompt tokens
Vision Transformer block integrated with VAPT
VAPT block
Figure I: Overview of VAPT. Unlike VPT, where prompt tokens remain constant irrespective of the input, VAPT dynamically generates prompt tokens p()) conditioned on the input X via a VAPT block. This endows prompt experts with more adaptive and expressive functional formulation.
that captured by pre-trained experts. To this end, our prompt experts are designed to extract global information from the feature map using token-wise projectors, defined as follows:
Gyr
Oy.AR E IRd
(8)
R=1
for E [Nr)], where a' are learnable scalars for & [N]. These projectors aggregate tokens across spatial locations, thereby facilitating global information exchange within the feature map.
Channel-wise Convolution. Token-win wise projectors aggregate features through weighted averaging of feature tokens. While computationally efficient, this approach may not fully capture important spatial relationships. For instance, the adjacency between 21 and Z2, which correspond to neighboring patches in the feature map, can be overlooked by token-wise projectors that treat tokens independently of their spatial origin. To address this, we introduce channel-wise convolution layer, applied the input feature map before token-wise projectors. Let F = [w [ represent a kernel of size K. Unlike standard convolution kernels which utilize distinct weights for each input channel, J reuses the same K x K weights across all d channels. This design significantly reduces the parameter count by a factor d, without sacrificing ability to model local spatial interactions. We show that this channel-wise convolution not only saves parameters but also improves performance in Appendix E.6 Formally, channel-wise convolution is applied to X imgg as:
X F X ing R
H' XX' xd
where denotes the 2D convolution operation, and H' = K and W = W - K+ are the height and width of the output feature map, respectively. By convolving neighboring patches, this operation explicitly encodes local spatial relationships into the feature map X Xoonv- We employ single channel-wise convolution layer within each ViT block. The resulting feature map Xconv is cony E H'' before being processed by the token- subsequently flattened to [æcoav, ..... #H' wise projectors. The final aggregated features can be expressed as:
H'WW" Gy = Σ
ay cony = WT X E Rd
(10)
k=1
for j' E [Np]. Crucially, since both channel-wise convolution and token-wise projection are linear operations, the aggregated features also constitute linear function of X, with the transformation weights W,, E RdxxNd, This aggregated information is then leveraged to construct our adaptive prompts in the next section.
4.2 ADAPTIVE PROMPT
Each prompt initially aggregates its corresponding global feature token Gy (X (Conv) through token- wise projector. Subsequently, generate the final adaptive prompts, we introduce a feature projector implemented as small MLP g R + Rd defined as follows:
g(x) (2) (W())))
(11)
5

---

Published as a conference paper at ICLR 2026
where W()) E Rxxd W(2) E Rdxr, < d, and ((.) is non-linear activation function (ReLU in this work). This feature projector is applied to the aggregated features to produce the final adaptive prompt tokens. Formally, the adaptive prompts at each block are given by:
(x) = g(Gy (X conv)) )) = W (w = w (w (1) X) E R",
(12)
for j' E [Nypl where w = Wy, Our proposed adaptive prompts refine the prompt experts and their associated score functions within the MSA layers as follows:
fN+3 (X) WM Px(
(13)
X E. WOWK m Px(
S1,N + (X)
(14)
where i E [M] and j' E [AN]]. Equation (13) demonstrates that the prompt experts become adaptive to the input X, enhancing their expressiveness and addressing the limited flexibility discussed in
Section The statistical advantages of this formulation are further analyzed in Section [
Efficiency Considerations. Following VPT, we insert adaptive prompts into every ViT block. While the feature projector is a lightweight MLP, distinct projector for each block would incur substantial parameter and memory overhead. To mitigate this, we employ a single, shared feature projector g()) reused across all VIT' blocks. This significantly reduces the overall computational cost and enhances efficiency, while preserving prompt expert expressiveness relative to VPT. Furthermore. an independent LayerNorm (Lei Ba et al.) 2016) is incorporated, we incorporate an independent LayerNorm (Lei Ba et al., 2016) before the token-wise projectors in each block. Regarding parameter complexity, VPT introduces approximately L x Np x d parameters across L VIT blocks. In contrast, VAPT's learnable parameters include token-wise projectors, a channel-wise convolution layer per block, and shared feature projector. The total parameter count is given by:
Lx x H' x W + L K2 2 2xrxd
(15)
token-wise projectors
convolution
fenture projector
where H1 x W < N, with K, r as small constants. Notably, for typical ViT configurations (e.g., ViT- B/16, N 196 and d 768). N is considerably smaller than d. Consequently, VAPT can achieve greater parameter efficiency than VPT, as empirically shown in Table Table Moreover, VAPT introduces only marginal computational overhead, up to 0.6% relative VPT (see Appendix E.7
Novelty. A central contribution of VAPT is its ability to achieve both flexibility and effectiveness while preserving computational efficiency. A common assumption is that increasing functional expressiveness necessarily incurs substantial computational or parameter overhead. As outlined in the above analysis, naive implementation of linear prompt expert would require prohibitively large number of parameters due to the high dimensionality (Nd) of the input X. In contrast, VAPT strikes a favorable balance between parameter efficiency and the expressivity of prompt experts. Despite the absence of principled framework for designing adaptive prompts, VAPT's architectural choices of token-wise projectors and channel-wise convolutions yield a straightforward implementation and compact, interpretable prompt-expert formulation. This architectural simplicity is particularly important, as it enables the rigorous theoretical analysis in Section 5 where we show that VAPT achieves optimal sample-efficiency rate. Prior work often relies more complex architectures or heuristic mechanisms to adapt prompts across inputs (Wang et al. 2022 Huang et al., 2023 Kim et al. 2023). While these designs can perform well empirically, their functional complexity typically precludes meaningful theoretical understanding, By contrast, VAPT is deliberately designed to retain simple yet expressive functional form, leading to the clean expert formulations in Equations (13 and (14). These formulations support a thorough theoretical analysis without sacrificing empirical performance, addressing a key gap in the literature: VAPT not only performs well in practice but also enjoys provable robustness and generalization guarantees. Importantly, our approach leverages the existing MoE structure implicit in attention heads rather than imposing an additional, external MoE module, as is common in prior work (Zeng et al. 2025). Conventional MoE-based methods explicitly insert new routing components and expert modules into the Transformer architecture, introducing substantial architectural modifications. In contrast, VAPT avoids such intrusive changes to the backbone while still admitting a formulation that is amenable to formal analysis. This perspective allows us to design VAPT as simple and practical mechanism that fully benefits from MoE theory to rigorously characterize its behavior, For a more detailed comparison with related methods, please refer Appendix C

---

Published as a conference paper at ICLR 2026
5 STATISTICAL ADVANTAGES OF VAPT
In this section, we explore the theoretical advantages of VAPT through its MoE connection, as detailed in Equation 7 This perspective provides rigorous framework for analyzing the convergence properties of prompt estimation in MoE models (Le et al. 2024 2025b). Recalling from Section 3 that the MoE models fm..:...,, hm.N . in each attention head share a common structure of experts and score functions. Furthermore, noted in Section 2.1 omitting fmm,N+ Nrn..N1......... hm. N-NN, does not affect ViT block output. Thus, to simplify our analysis while maintaining rigor, we focus on the first head (i.e., 1), and specifically the first row of its attention matrix (i.e., i 1) in Equation 7 Within this simplified setting, we consider a regression framework for MoE models as follows.
Problem Setup. Assume that the i..d. samples of size (X1, (22, (.). Rd Rdd are generated from the following regression model:
Y fa. (X,, t i= 1.2.......
(16)
where S1, E2,...,... En are independent Gaussian noise variables with [X, = 0 and Var(e, = 2² La for i E [n]. Furthermore, X1, X2...., Xn, are assumed to be i..d. samples from distribution u. The ground-truth regression function fa. (.) is an MoE model of the form
N exp( X 14X + +a))
[x. =
1(X, n))
DJ.G((X) (X)
2,200 (W.,13 x)) X + bun')) CW (W.12 X),
(17)
(X)
===1
where D,,G. (X) = En= exp(X A4 - a)) + Σド=
+ be.r).
Here, G. = Σラ E j6w.., 1y..,,,... ..)) denotes the true mixing measure, which is a weighted sum of Dirac measures 8 associated with the unknown parameters (bx.z", W.,13', 2 in the parameter space θ C x Rexd x Rdxr. The matrix AS. the expert parameter v7.. and the bias parameter at are known for j E [N]. Finally, the matrices B E Rdxd and C E Rd'xd are given; these play the role of pre-trained projection matrices in the context of prompt tuning.
Least-Squares Estimator: To estimate the unknown prompt parameters or, equivalently, the ground- truth mixing measure G., we use the leaat-squares method (van de Geer 2000). In particular, we consider the estimator defined as follows:
G == arg wig.ane (0) Σ Y; - (o-=f((x)))""
(18)
[L', (b,, W1.4, W2) E }} is the set of all mixing measures with at most L' atoms. Since the true number of experts L is generally unknown, we assume the number of fitted experts L' is sufficiently large, i.e., L' > L. To analyze prompt estimation convergence, it is essential to define a suitable loss function on the prompt parameters. In this work, we propose the Voronoi loss function, derived from the concept of Voronoi cells (Manole & Ho. 2022).
where (L) = (G
Voronoi Loss, Given a mixing measure G E GZ,( consider Voronoi cell set (V/ Vj(G), E [ generated by the atoms of G., where
V, == {i [L']: ||, Z,, || S || Z. 1l. #}.
where Z, = (Wi.i, W2). The Voronoi loss tailored to the setting in Equation is defined as:
(G, Σ
Σ VEV
exp(b:) exp(b.,;))
Σ
Σ
exp(b:) ((Wa))
J'ELLHV y --y
Σ
Σ
exp(6))(IAW 11AWs1)),
7

---

Published as a conference paper at ICLR 2026
Table 1: Overall Comparison for ViT-B/16 Supervised Pre-trained on ImageNet-2IK. Following Jia al. (2022). we report average accuracy (3 runs) on FGVC and VTAB-IK, "Number of Wins" [-] compared to full fine- -unittnnng..a and "Number of Wins over VPT {.). "Tuned/Total" denotes the average percentage of parameters tuned (24 tasks), "Scope" specifies the tuning scope, and "Additional parameters" indicates if parameters beyond
pre-trained backbonc/head are introduced. Per-task results in Appendix T.
(Dosoviskiyy
Tuned/ (08a)(55)
Ingut
Scope Backhone
Extra params
RGVC|5
Natural [7]
VTAB al. 2019/
Specialis/1 [4]
Strecturd [8]
Mean Tota
Foll lotinova of al.
2022
100.00
58.54
75.88
83.36
47.64
65.57
Linear Rolnne ti 2019
0.08
79.32
68.93
77.16
26.84 [0]
52,91
Partil-
2014
8:34
82.63
69.44
78.53
34.17 [0]
1.44
79:80
67.80121
72.83
30.62 [0]
53.21
Sidetue Zhang al. 2020
10.08
/
78.35 [0]
58.21 [0]
08.12 [0]
23.41 [0]
45.65
Bias [Rebin Mille 13:
0.30
$8.41 [3]
[3]
78.25 [0)
44.09 [3]
62.05
Adapter me B034
1.02
85 46
70.67 [4]
77.80
33.09 [0]
62.41
LoRA (iirecii
0.73
89:46
78.26
63.78
56.20 [7]
72.25
2022
0.16
84621111
79.66
[4]
VPT-Deg
0.73
89
48.48 [6]
82.43
34.98 [8]
69.43
E2VPT 2f 20z4
0.39
50.22 [4]
80.01 [6]
84.43 [5]
$7.39 [8]
71.42
SA2VP IVEEU PUM4
0.65
90.08 [4]
$0.97 [6]
65.73 [4]
60.50 [8]
73:47
VaPT Ai
0.66
91 40 [4]
82:63 [6]
85.22
61.25 [8]
73.70
VEPT
0.66
89:24
81.35
$4.93
60.19
73.20
VAPT (Oues)
0.36
89.58 [4)58 (4)
81.43 (6) (?)
85.13 (4) (4)
59.34 [8]
72,91
where we denote AW1y W1+ Wyy for any j, and AW2 := W2 W
Equipped with this loss function, we wrap up the setting in Equation (17) by providing the convergence rate of prompt estimation in Theorem[] For that purpose, it is necessary to make essential assumptions on the activation function a. However, due to the space limitations, we defer these assumptions to the proof of Theorem i Appendix [
Theorem 1. Let GR be the least-squares estimator defined in Equation (8) and assume that the
activation function a satisfies the Assumptions (A.1)-(A.3) specified in Appendix A. we obtain that
Di,G,, G.) (po(((())//))))
Implications for Prompt Estimation. Given the formulation of the Voronoi loss function D1. Theorem ! indicaatss that the estimation rates for the true parameters (Wa,15s W..2 for indices j such that 1yl 1 are of parametric order ((oog(n)//]]3 Due to the Lipschitz property of the activation function a, this directly leads to estimation rate of Op(loo(n)///) for the true prompt P, (x) = W_ On the other hand, for true parameters (W.,u W.,2) where which yields an estimation for true prompt P. (X) = W go (Wr,1 Finally, all these rates are optimal, up to the logarithmic factor, demonstrating the statistical benefits of visual adaptive prompt tuning for the non- non-linear setting of the activation function σ.
131 > 1, their estimation rates of the order Or((lo(n)///)) rate of ([og((()))///)
Linear Activation Setting, For completeness, we also show that the visual adaptive prompt tuning achieves optimal sample efficiency when the activation function a(-) linear identity in Appendix B
EXPERIMENT
In this section, compare VAPT with VPT and other widely used PEFT methods. We also examine the robustness of VAPT under different pre-training objectives and present our findings on its sample efficiency. For additional results, including ablation studies, semantic segmentation results, statistical significance tests, computational cost, and interpretive visualizations, please refer to Appendix F
6.1 EXPERIMENTAL SETUP
Datasets. We evaluate VAPT on two benchmarks: FGVC and VTAB-1K Zhai et al. 2019). FGVC includes five fine- grained classification datasets: CUB (Wah et al.) 2011). Oxford Flowers (Nilsback) & Zisserman 2008), Stanford Cars (Gebru et al., 2017), Stanford Dogs (Khosla et al. 2011). and NABirds (Van Horn et al. 2015) that require distinguishing between visually similar classes VTAB- IK comprises 19 datasets, each with 1,000 training examples, grouped into: Natural (images captured by standard cameras), Specialized (images collected via specialized equipment), and Structured (tasks requiring structural understanding, such as 3D depth prediction). Beyond classification, we also assess performance on semantic segmentation using ADE20K (Zhou et al. 2019).
8

---

Published as a conference paper at ICLR 2026
Table Comparison of Different Pre-training Objectives. We consider MAE (He el. 2022 and MoCo
v3 (Chen et al. 2021) using ViT-B/16. We report test accuracy on VTAB- IK the "Number of Wins" [-] relative to full fine-tuning tuning and the "Number of Wins over VPT" (.). "Tuned/Total denotes the average percentage of parameters tuned. Per-task results are in Appendix
Pre-trained objectives
MAE Ho 2022
MoCo v3
Chhn al 2021
Params & Data
Tuncd/
VTAB-IK
Zhai et al. 2019
[19]
Tuncd/
VTAB-IK K
Zhai at 2019
[19]
Methods
Sprcimlterr
Total (9))
Natural [7]
Specwwrzed [4]
Streetwee [8]
Full Lofinova et al. 2022
100.00
59.31
79.68
53.82
100.00
71.95
84.72
51.98
Linear (odinova et 2022
18.87
53.72
23.70 [0]
30.33 [0]
Partial- TTsmiisk of
8.30
58.44 [5]
78.28 [1]
47.64 [1]
8,30
72.31 [5]
84.58 [2]
47,89 [1]
(Rebaffi al 2017
54.55
75.68
47.70 [0]
72.89
81.14
53.43 [4]
Adapter (Ca al 20230
0.87
54.90 [9]
75.19 [1]
38.98 [0)
1.12
74.19 [4]
82.66 [1]
47.69 [2]
VPT-Shallow Dia al 2022
0.05
39.96
69.65
27.50
0.06
67.34
82.26
37.58 [0]
Derp Dii et pizza
0.31
36.02 [0]
60.61 [1]
26.57 [0]
0.22
70.27 [4]
83.04 [0]
42.38 [0]
GateVPT Yrnrel mis 0124
0,05
47.61
76.86
36,80 [1]
01.06
74.84
83.38
49.10 [3]
E2VPT (H
59.52
77.80
[6]
ViaPT Niao 2025
0,36
34,26 [-]
78.01 [-]
37.52 [-]
0.30
79.12 [-]
$6.81 [-]
60.05 [-]
VFPT
Zong 2022
0.38
53.59
77.75
36.15 [1]
0.22
77:47
85.76
58.74 [6]
VAPT (Ours)
0.28
59.23 (5) (7)
80.73 [2] (3)
47.24 (2) (73)
0.27
77.69 [6] (7)
83,95 (2) (3)
60.74 (7) (8)
Baselines. In line with prior work (Jia al. 2022 Han al. 2023), we compare VAPT against
commonly used PEFT methods. All classification experiments use standard Vision Transformer (ViT) (Dosovitskiy 2020) that are supervised pre-trained on ImageNet-21K: (Deng et al.) 2009). For semantic segmentation, we employ SETR-PUP (Zheng et al., 2021), which uses ViT as the backbone encoder. Additionally, we evaluate VAPT with backbones pre-trained using two self-supervised learning: MAE (He et al. 2022) and MoCo v3 (Chen et al. 2021).
Training. We perform a grid search on val set of each task to determine the optimal learning rate, weight decay, kernel size K, and projector dimension r. We schedule the learning rate using a cosine
decay schedule and train all models for 100 epochs. Following Jia et al. (2022), we use batch sizes of 64 and 128. Additional implementation details can be found in Appendix
6.2 EMPIRICAL RESULTS
Overall Comparison. Table 1 compares VAPT against full fine-tuning and other prominent PEFT methods on VTAB-IK and FGVC. Full denotes full fine-tuning, which updates all model parameters, while methods such as Linear, Partial-1 (top layer), and MLP-3 (3 MLP layers) modify only a subset
of parameters. Sidetune (Zhang et al. 2020). Bias (Rebuffi et al. 2017). Adapter (Cai et al.) 2020),
and LoRA (Hu et al.) 2021) introduce trainable modules for adaptation. Concurrent visual prompt tuning approaches, VPT Tia et al., 2022). E2VPT (Han et al. (023). SA2VP (Pei et al. (024).
ViaPT (Xiao al., (025) and VFPT Zeng et al., 2024). are also included for comparison (see Appendix Cor further details). Our results indicate that VAPT surpasses full fine-tuning in 22 out of 24 tasks. Specifically, VAPT achieves a notable 1.04% % accuracy increase on FGVC and a substantial 11.70% % improvement on VTAB-IK Structured. Across the entire VTAB- IK benchmark. VAPT demonstrates an average gain of 7.34% over full fine-tuning, while updating merely 0.36% of the backbone parameters. These findings highlight VAPT's significant effectiveness and efficiency as an innovative PEFT method. Additionally, VAPT achieves state-of-the-an performance compared to other PEFT approaches. Among prompt tuning methods, VAPT consistently outperforms VPT in 23 out of 24 tasks and attains competitive performance with VFPT, a recent state-of-the-an approach, despite utilizing nearly 50% fewer parameters. We attribute these improvements VAPT's design, which enhances the expressiveness of prompt experts. These results underscore VAPT's potential as a powerful tool for improving performance with significantly reduced parameter overhead.
Different Pre-training Methods. We investigate VAPT's performance when initialized with back- bones pre-trained using different self-supervised learning (SSL.) objectives. specifically MAE (He 2022) and MoCo v3 (Chen et al. (2021). with detailed results in [ Previous studies (Jia ct al. 2022: Yoo et al., 2023) have indicated that VPT can exhibit suboptimal performance with SSL pre-trained backbones. In contrast, VAPT achieves significant performance improvements by effectively leveraging the rich information present in the input features. For example, VAPT demonstrates a remarkable 23.21 % accuracy improvement under MAE VTAB-IK Natural tasks and an 18.36% % improvement with MoCo v3 on VTAB- IK Structured. Furthermore, when compared to other PEFT methods, VAPT consistently outperforms them, achieving the highest "Number of
et al.

---

Published as a conference paper at ICLR 2026
Wins" relative to full fine-tuning, with 9 out of 19 tasks under MAE and 15 out of 19 tasks under MoCo v3. In addition, VAPT outperforms state-of-the-ee approaches such as VFPT under MAE and remains competitive under MoCo v3. These results underscore the generality and robustness of our method across different pre-training objectives, supported by both theoretical analysis (Section and empirical evidence.
Sample Efficiency. To empirically validate the theoretical sample efficiency from Section [5] we conducted experiments on the Stanford Dogs dataset (Khosla al., 201T). Following Ascoli et al. )021). we subsampled each class by frac- tion (0.01, 011, 0.3, 0.5, 1.0) and scaled the number of training epochs by 1/f. thereby ensuring constant total number of image pre-
Acouracy (%)
sentations to the model. The results, presented in Figure and Table 3 demonstrate that VAPT consistently outperforms VPT across all evalu- ated training set sizes. Notably, with only I 1% of the data, VPT achieves accuracy of 3.6 %,, whereas VAPT attains a substantially higher ac- curacy of 60.1% %. Furthermore, VAPT requires only 30% of the data to match the performance of VPT trained the full 100% dataset, signify- ing an approximate 3x reduction in data require- ments. This underscores the superior sample efficiency of our approach.
Metted
WAPT
12"
16-1 traction (log scale)
Figure Comparison VAPT VPT across varying fractions of training data on Stanford Dogs.
Table Classification accuracy on Stanford Dogs (Khosla et al.) (011) using varying fractions of training data. Bold indicates the best performance.
important note that the primary objective of this work is not to establish a new state-of- the-art on every dataset. As outlined the ab- focus is scholarly
Training fraction
VPT
VAPT
Gap
stract and introduction, value rather than purely incremental method- ological gains: we aim to provide the commu- nity with clear insights into the benefits of in- creasing the functional expressiveness of prompt experts within visual prompt tuning frameworks. VAPT serves as a concrete instantiation of this idea. Its adaptive formulation illustrates the potential of adaptive prompt experts to deliver improved performance, stronger parameter efficiency, and enhanced sample efficiency both theory and practice.
1%
3.6
60.1
56.5 %
10%
83.3
88.0
4.7%
30%
87.8
90.3
2.5%
50%
87.5
90.9
3.4%
100%
90.2
91.7
1.5%
Within this perspective, the most relevant comparison is between VAPT's adaptive prompt-expert formulation and VPT, where prompt experts are constant functions. Our experiments consistently show that VAPT outperforms VPT on the majority of benchmarks while using a more compact set of trainable parameters. Moreover, our empirical findings quantify the statistical advantages VAPT over VPT, and these observations are rigorously supported by our theoretical analysis in Section 5 and Appendix [A] Together, these results reinforce our central claim that increasing the expressiveness of prompt experts is principled and effective direction for advancing visual prompt tuning.
7 CONCLUSION
In this paper, we highlight the limited functional expressiveness of prompt experts in existing VPT formulation and introduce VAPT, novel adaptive prompt design. Our approach achieves superior performance compared to VPT while using fewer parameters and demonstrates optimal sample efficiency both theoretically and empirically. Our theoretical analysis is grounded in interpreting self-attention as composition of multiple MoE models. Given that self-attention is central to most Transformer architectures, we believe our analysis can naturally extend to other Transformer variants. While our experiments were conducted Vision Transformers, the potential demonstrated by adaptive prompts suggests that future work could explore alternative designs across diverse Transformer architectures and broaden their applications to wider range of tasks. Furthermore, future research could investigate the potential synergies between VAPT and VPT, for instance, by exploring their integration.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
In order to facilitate the reproduction of our empirical results, we provide detailed descriptions of the experimental setup in Section 6. and Appendix [ available, enabling full replication of our experiments.
All datasets used in this study are publicly
REFERENCES
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, al. Deepmind lab, arXiv preprint arXiv:1612.03801, 2016.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances Neural Information Processing Systems, 33:11285-11297, 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint NiKiv:2003.042990 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9640-9649, 2021.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10): (115501):865-18880 1883, 2017.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009.
Nghiem Diep, Huy Nguyen, Chau Nguyen, Minh Le, Duy MH Nguyen, Daniel Sonntag, Mathias Niepert, and Nhat Ho. On zero-initialized attention: Optimal prompt and gating factor estimation. arXiv preprint arXiv:2502. 03029,, 2025.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arX1v:2010.11929, 2020.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-ofeexperts. In International Conference on Machine Learning, pp. 5547-5569. PMLR, R, 2022.
Stéphane d' Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International conference on machine learning, pp. 2286-2296. PMLR, 2021.
D. Eigen, M. Ranzato, and I. Sutskever. Learning factored representations in deep mixture of experts. In ICLR Workshops, 2014.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (clip). In International conference on machine learning, pp. 6216-6234. PMLR, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23( 1-19))11-3,, 2022.
Li Fei-Fei, Robert Fergus, and Pictro Perona. One-shot learning of object categories. IEEE transac- fions on pattern analysis and machine intelligence, 28(4):594-61 611, 2006.

---

Published as a conference paper at ICLR 2026
Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine- fine-grained car detection for visual census estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics; The kitti dataset. The International Journal of Robotics Research, 32(11), 2013.
Ben Graham. Kaggle diabetic retinopathy detection competition report. University of Warwick, 22 (9), 2015.
Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E" 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.13770, 2023.
Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, and Dongfang Liu. Facing the elephant in the room: Visual prompt tuning or full finetuning? arXiv preprint arXiv:2401.12902, 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards unified view of rrrameeer--fffcien transfer learning. arXiv preprint 0rXiv:2110.04360 2021.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter-efficien model adaptation for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 817-825, 2023.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-22266 2019.
Nhat Ho, Chiao Yuiao-YuPoe Yang, and Michael Jordan. Convergence rates for gaussian mixtures of experts. Journal of Machine Learning Research, 23(323):1 81, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhuu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 000. 2021.
Lingyun Huang, Jianxu Mao, Yaonan Wang, Junfei Yi, and Ziming Tao. Cvpt: visual prompt tuning adapt visual task. arXiv preprint ar Xiv: 14961, 2024.
help
Qidong Huang, Xiaoyi Dong, Dongdong Chen, Weiming Zhang, Feifei Wang, Gang Hua, and Nenghai Yu. Diversity-aware meta visual prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10878-10888 2023.
Eugenia lofinova, Alexandra Peste, Mark Kurtz, and Dan Alistarh. How well do sparse imagenet models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12266-12277 2022.
R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3, 1991.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
12

---

Published as a conference paper at ICLR 2026
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-2910, 2017.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures experts and the em algorithm. Neural computation, 6(2):181-214 1994.
Aditya Khosla, Nityananda Jayadevaprakash. Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proc. CVPR workshop on categorization (FGVC), volume 2. Citeseer, 2011.
visual
Doyoung Kim, Susik Yoon, Dongmin Park, Youngjun Lee, Hwanjun Song, Jihwan Bang, and Jae-Gil Lee. One size fits all for semantic shifts: Adaptive prompt tuning for continual learning. arXiv preprint 1.Xiv:23111120444 2023.
Youngeun Kim, Jun Fang. Qin Zhang, Zhaowei Cai, Yantao Shen, Rahul Duggal, Dripta S. Raychaud- huri, Zhuowen Tu, Yifan Xing, and Onkar Dabeer. Open-world dynamic prompt and continual visual representation learning. In European Conference on Computer Vision, pp. 357-374. Springer, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4015 4015-4026, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Minh Le, An Nguyen, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Van Ngo, and Nhat Ho. Mixture of experts meets prompt-based continual learning. Advances in Neural Information Processing Systems, 38, 2024.
Minh Le. Tien Ngoc Luu, An Nguyen The, Thanh- Thien Le, Trang Nguyen, Thanh Tung Nguyen, Linh Ngo Van, and Thien Huu Nguyen. Adaptive prompting for continual relation extraction: A within-task variance perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2025a.
Minh Lc, Chau Nguyen. Huy Nguyen, Quyen Tran, Trung Le, and Nhat Ho. Revisiting prefix- tuming: Statistical benefits of reparamcterization among prompts. The Thirteenth International Conference Learning Representations, 2025b.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):228--3224, 1998.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. 11-104. IEEE, 2004.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv e-prints, pp. a6Xiv-1607, 2016.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv: 12..1004..1, 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706.66083, 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018.
13

---

Published as a conference paper at ICLR 2026
T. Manole and N. Ho. Refined convergence rates for maximum likelihood estimation under finite mixture models. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14979-15006. PMLR, 17-23 Jul 2022.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised featue learning, volume 2011, pp. 4. Granada, 2011.
Huy Nguyen, TrungTin Nguyen, and Nhat Ho. Demystifying softmax gating function in gaussian mixture of experts. Advances in Neural Information Processing Systems, 36:4624-4633 2023.
Huy Nguyen, Pedram Akbarian, and Nhat Ho. is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICMI., 2024a.
Huy Nguyen, Nhat Ho, and Alessandro Rinaldo. On least square estimation in softmax gating mixture of experts. In Proceedings of the ICML, 2024b.
Huy Nguyen, Nhat Ho, and Alessandro Rinaldo. Sigmoid gating is more sample efficient than softmax gating in mixture of experts. In Advances in Neural Information Processing Systems, 2024c.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Imagge Processing. IEEE, 2008.
Samet Oymak, Ankit Singh Rawat, Mahdi Soltanolkotabi, and Christos Thrampoulidis, On the role of attention in prompt-tuning. In International Conference on Machine Learning, pp. 26724-26768, PMLR, 2023.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEFE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Wenjie Pei, Tongqi Xia, Fanglin Chen, Jinsong Li, Jiandong Tian, and Guangming Lu. vp: Spatially aligned-and add-td visual prompt. In Proceedings of the AAAI conference artificial intelligence, volume 38, pp. 4450-4458, 2024.
Aleksandar Petrov, Philip HS Torr, and Adel Bibi. When do prompting and prefix-tuning work? a theory of capabilities and limitations. arXiv preprint arXiv:2310. S.Xi:23310.196999 196998, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 7748-8765 PMLR, 2021.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances neural information processing systems, 30, 2017.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems. 34:8583-8595 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
14

---

Published as a conference paper at ICLR 2026
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The aparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. arXiv preprini arXiv:2303.07226. 2023.
Kihyuk Sohn, Huiwen Chang, José Lezama, Luisa Polania, Han Zhang, Yuan Hao, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19840-19851, 2023.
S. van de Geer. Empirical processes in M-estimation. Cambridge University Press, 2000.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pp. 210-218. Springer, 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Yihan Wang. Jatin Chauhan, Wei Wang, and Cho-Jui Hsieh. Universality and limitations of prompt tuning. Advances in Neural Information Processing Systems, 36, 2024.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022.
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution, pp. 196-202. Springer, 1992.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to z0o. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang. Yuxiang Wei, Jihun Hamm, and Min Xu. Visual instance-aware prompt tuning. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 1880-2889 2025.
Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-effiient fine-tuning for pre-trained vision models: A survey. arXiv preprint arXiv:2402.02242, 2024.
Lei Xu, Michael Jordan, and Geoffrey E Hinton. An alternative model for mixtures of experts. Advances in neural information processing systems, 7, 1994.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-guud guided context optimization. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 6757-6767, 2023.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision- laiguaee models. Al Open, 5:30-38, 2024.
Seungryong Yoo, Eunji Kim, Dahuin Jung, Jungbeom Lee. and Sungroh Yoon. Improving visual prompt tuning for self-supervised vision transformers. In International Conference on Machine Learning, pp. 40075 PMLR, 2023.
15

---

Published as a conference paper at ICLR 2026
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances neural information processing systems, 27, 2014.
Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huang, Ying Nian Wu, and Dongfang Liu, Visual fourier prompt tuning. arXiv preprint arXiv:2411.0132) 2024.
Runjia Zeng. Guangyan Sun, Qifan Wang. Tong Geng, Sohail Dianat, Xiaotian Han, Raghuveer Rao, Xueling Zhang. Cheng Han, Lifu Huang, et al. Mept: Mixture of expert prompt tuning as manifold mapper. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 16271-16291, 2025.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv 2019.
preprint
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: baseline for network adaptation via additive side networks. Computer Vision-ECCH 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 698-714. Springer, 2020.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip HS Torr, al. Rethinking semantic segmentation from sequence-to-sequence perspective with transformers. Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 6881-6890, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-33 321, 2019.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference computer vision and pattem recognition, pp.
2022.
Yanqi Zhou, Nan Du, Yanping Huang, Daiyi Peng, Chang Lan, Da Huang, Siamak Shakeri, David So, Andrew M Dai, Yifeng Lu, et Brainformers: Trading simplicity for efficiency. International Conference on Machine Learning, pp. 12531-42542. PMLR, 2023.
16

---

Published as a conference paper at ICLR 2026
Supplement to "Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts"
In this supplementary material, we provide detailed proofs of the main results in Appendix A and additional theoretical findings in Appendix B A discussion of related work is presented in Appendix C Implementation details of our experiments are described in Appendix while additional experimental results are included in Appendix E Specifically, these include detailed per-task results corresponding to the main experiments in Section 6
(Appendices E.I E.2). statistical significance tests (Appendix F.3), an evaluation of VAPT performance across different backhone scales (Appendix [..4 and semantic segmentation results (Appendix (.5)). Finally, ablation studies, computational cost analyses, adversarial robustness, and interpretive visualizations are presented in Appendix E.6 Appendix E.7 Appendix E.8 and Appendix E.9 respectively.
A PROOFS
In this appendix, we provide proofs for key theoretical results on prompt estimation presented in the main text.
A.1 PROOF OF THEOREM
Proposition 1. Given the least-squares estimator Gn defined in Equation (8). under the T22(1) norm, the estimator fa. (.) converges to the true model (.) at a parametric rate with respect to the sample size. That is,
llfe - Sc. leosew = ((((oo(m))/n]))
(19)
The proof of Proposition iss provided Appendix A.2 Building on the convergence rate established therein, we now aim to demonstrate the following inequality:
inf (e)
Ilfc - fc. /D.ll G.) 0.
(20)
We divide the proof of the above inequality into local and global parts presented in Appendices A.I.] A.112 respectively. Before presenting the proof details, let us introduce some essential assump- tions on the activation function
Assumptions. We impose the following assumptions on the activation function a:
(A.1) (ldentifiability) If there exist parameters (W1, W2) and (W1, W2) such that W20 WX holds for almost surely X, then (W1, W2) = (W1, W2).
(A.2) (Uniform Lipschitz) Let F(X; W1, W2) exp( W4 ()) XW).). Then, for any 5 E {1, 2}. we have
gla F aWai awa (X W1, W2)
alal F
(X W1, (:W W3) < CII(W,, (2) (W1, WDI
a aw
la|=T
for any vector Y E 2dr and for some positive constants Ç and C' that are independent of X and Here, a = (a1, E Noxd x Ndxrr
(W1, W2),
(A.3) (Strong identifiability) The function is twice differentiable almost surely. For any natural number l and distinct parameters (Wi. j E [(]) the functions in the set
(((1,
(1) (Wi, X) (w) (1) XOo
x(e)
(VV
X
(W1 ( (W1, x (v) (w))
( (w) (2) (W X) X (1) (())
22) ( Xxx j E 10 u, v w [d]
are linearly independent for almost surely X. Here, (1) and (2) denote the first and second derivatives of a, respectively, which applied element-wiss to the matrices W1, X.
17

---

Published as a conference paper at ICLR 2026
A.1.1 LOCAL PART
The local part of the inequality 20 corresponds to the following inequality:
[l - Sa. IlLa(w) O. 5 GE PIC,c.) G.)
lim
inf
We assume, for the sake of contradiction, that the above inequality does hold. Then, can find sequence of measures Gn == Σÿ= exp(bn (d(.) .W..)) in (O) such that
Wr.2)
Din == Di G-) +0 { U1 JG.llia(u)/Din DIn + 0.
For the sake of the presentation, V == Vj(G,,) is denoted Voronoi cell of G, generated by the j-th components of the true measure G.. Given that the ensuing arguments are asymptotic, without loss of generality we assume that those Voronoi cells do not depend on the sample size, i.e, we have Vj- = V for all n and S j S L. Hence, the Voronoi loss Din can be rewritten as follows:
Din
(ex((i))eep(()))))) 5'=1 SVVY
Σ
E
| exp(b,,.)
E
(,W,,,)
Σ (E[4HIyybIEEV
Σ
exp(In.)(IIAW.
||W n,
where we define AW, Wn,14 W and AW n.2 W W. for all Vyr
From the hypothesis, since Dn - as n + oc, we have Eiev, exp(bn.;) - exp(b..), Wn,1i W,,,'' and W,,2 + Wa,2 for any E Vj.j - [4]. Throughout this proof, for simplicity of argument we assume without loss of generality that B Ia, = Id. and r 1. We note that our techniques can be generalized to the general case of these given matrices. Now, we divide the proof of the local part into the following three main substeps:
Step 1 Taylor expansion.
We first define the following function:
N Σ
Qx(X)
exp
X
A x
X x+brr") [sa, (X) (X)].
J=l
f'=1
Then, we can decompose the function (X) as follows:
Qn(X) Σ
exp(bmni)
j= EEV
-exp((W.,oW...xxX) ()))))))),.0000 X)
exp(tm.))
- x) fa. fx, {x.)) (x)
jelie,,
exp(bn.i) - exp((bxs) exp((W, x) W. WW Sc. (x))
j=1
SEV;
A. X (x) + (X).
(21)
Decompasition of the function (X). To streamline the argument, we define the following func- tions: É(X; W1, W2) = e more, the product of these functions is defined as F(X;W; W2) = ( W,. W. (X; W2). To account for the difference in the number of elements among Voronoi cells, we further decompose
X) and H W.,WW, We (W, Further-
18

---

Published as a conference paper at ICLR 2026
the function A, (X) as follows:
An(
Σ
Σ exp(lmni) (F(;; Wn,2) F(X; W., 1,1.. Wo,2)
+
exp(bn.)) F(X;Wa,i Wn,2) F(X; W. W.,2)
= Am, (X) + (X)
An application of the first-order Taylor expansion leads to:
a E
E(X:Wn,1d Wn,2) E(X; F(X;W,,,, W.,15, W.,2)
("
(:; W.,jy Wa,2)
ow OW2
a
(x),
8 0 (X; W. W,, W.,2)
(X;:Wn,.. w222 = H(:: Wa,1s, 2o.2)+ + Σ (AWn,g))
Wwr1ow,"
|0|=1
(X).
Here, the indices j.j in the above equations satisfy [V1 = 1, i.e., Voronoi cells with exactly one element and E Vj. Furthermore, the functions Rx,,(()) and (x) in these expressions represent the Taylor remainders from the expansions of the functions E and H. Combining these results leads to:
exp(bna)) a!!
(.W,, 02 Hlarl-aaz1E (X; W. (X))) W.,,)
An,
Σ
Σ la|=1
a. aW
#|Vy/=liEM EEV
alail+aa2 H
+ (AW,, n16j on (A
[XW W, W. B(,; W W..1,, W. W..),, + (X)
a aW
glail+aaa1 E
Σ Σ
M Majaanas
(X; W. HH W..1j; Wa,2)
OW Ow
--1/==1 |a|=1
+ Ma.ja100 glai1+|aa1 H
X; W. B(,,) W..1,, Wo2) + (X)
oW OW?
where a = (a1, 02). Furthermore, due to the uniform smoothness of the functions E and H, the function (X) in the above equation satisfies R (x,)(XX) - 0 when approaches infinity. Finally, the terms Mnjjana in this equation admit the following forms:
exp(bm.i) a!
Maj.cnn
11) (AW,,,)
NEV,
for any Jai|= =
We now proceed to decompose the function 2,,((X) (X). Unlike the function A for which we utilized only a first-order Taylor expansion, the analysis of 2, must account for Voronoi cells potentially containing more than one element. Consequently, we employ second-order Taylor expansions of the functions E and Ht. In particular, an application of the second-order Taylor expansion leads to:
glanl+|aal E
An = Σ Σ
M Ma.i.on.o 02 OW OW?z (X: K.W, W. ((x W..)
HES 52
alaal+/oal H
+ Majaan Wo OW
X: W. 1j: W. F X; W ..jj, W.,)
ala E
g101|+ H
M, Mn.j...8
(X: W 1jj W.
(X; W 1y. W. + Rx)((X)
Ow ow
ow ww
|a/=1,
19

---

Published as a conference paper at ICLR 2026
where 02), 3 (31, B2). Furthermore, due to the uniform smoothness of the functions E and HT, the function (X), which represents the combination of Taylor remainders from the second-order Taylor expansion, satisfies coefficients Mn and M2. in the above formulation take the following forms:
0 as goes to infinity, Additionally, the
exp(bn.i) a!
Maj.ora22
11y (AW,..))
KEV,
for any multi-index a such that |a| and
exp(bn.s)
(AW (AW,))
a!!!
TE
for any coefficients a and such that |a] = |3] 1. Given the expressions for the functions
E( X: W1, W2) and (X;; W1, W2), their partial derivatives take the following forms:
0E
[: W1, W2) =xx(((>>((WWx)) X)X x (W)
(w (u)
aE
[X; W1, =2) (x)()))))x((( x) (0) x(Wi)),
aw!o))
X:: W1 = (x) W2 (W) (()
aw(") aw!))
(V)) W (2) (W, () X x())
PE
W;W, [X
aw!") ow!
PE
XX: (X:W, W2) <xp((W> X)X W20 (WI (WIx)
aw (1) !w
oxp((W2o(W))) x x(1)
aH
(X: W1, Wº) = W20 (W)
(
(X; W1, W2) = oWa
aw2
H
(X: WT W2) 12) (W)
ow(")ar (w(
H
(X: W1 W2) (x)x la.
ow!")aw aw,
82H
(X: W,, W2) O.
20

---

Published as a conference paper at ICLR 2026
Putting the above formulations together, the functions (x) and (x,2(X) can be rewritten as follows:
(x, =
Lnn.,......
+
(W.,,
XW. + o(W., X) + Rm,1 (X);
(n,((X)=
Σ
X)X Un.1..W...
XW. (W
+ (W + (W
X) Ln xX W.,2W.,2
XW. XW. (W X) X)X L.A...W...
X
+ (o"' x)] W. 2, + ( X Im,,6. XW.,
+ +o T.n + (o (W
Inn XX W.,2W..2
+
" 2 L, + o (o Ln.6.
+ x] x Lnno.al + (X),
where the formulations Im,1.j. Im.........
Tn,o in these equations are given by:
L Mn 02 2
Masiera = (Mn.jien.
Ln = (Mms.anew)un
Ln,3,3 Mndierada
Ln,A. = (Madeetee 02 0e),
Ln,5.j = (Mnj.Onentenn tew
Ln,6,J = (Mn.eewee
Here, for each 1 < S d, e, denotes the standard basis vector in Rd with 1 in the w-th position and O in all other positions.
Decomposition of the function B,(X) Similar to our decomposition of the function A, A,(X). in decomposing the function (x))), also distinguish between Voronoi cells containing exactly one element and those containing more than one element. Therefore, we obtain:
(x)
exp(bm.)) ii
Σ
Σ
E(X; Wn,16 Wn.2) E(X; W Wo.2) fc., (X)
Σ
Σ
exp(bm..) B(X; Wn.i Wn,2) ((X; W. W..2 Fc, (X)
> VEV,
= Bn, (X) Bn,((X).
For Voronoi cells with exactly one element, we utilize the first-order Taylor expansion, while for those with more than one element, the second-order Taylor expansion employed. This strategy leads to the following representations:
glan|+oall E aw aWas
(x,( = Σ
Σ
M Majjai.
(X; W,,, W..2)aa. (X) + Rx,((X),
H& |||=1
alan Haal E OWa Ow
(x,((X)= = Σ Σ
Mnjan
(X; W., W.,2) (X) - (X).
In these expressions, the functions R, X) and (x) correspond the Taylor remainders. Due to the uniform smoothness of the functions F,, we obtain Rn. X)/Din 0 and (X)
21

---

Published as a conference paper at ICLR 2026
approaches infinity. By computing the closed-form expressions for the partial derivatives of the function E, both functions (x,1(X) and (,, can be rewritten as follows:
(n,1(X)= = Σ
exp((W.2(WW.1xx3) x) (W Lnn + oo(W.,)))4n.2 (cn
/11/==1
+ (
(x)
L,a t (,,,, I. 2
/-12/121
+ (lol) X) + (W,, 15 L W., + (Wa,1j L. X
(W,,
W. + o (W.,
x)) (
Ln fc. (X) a
Combining these results, the function (x( can be rewritten as follows:
Qn(X)= = Σ ",, XW,,
=1
+
XW
XW. t ( 1
+ Σ
(W L Lm11
JI133 >1
XW, o(W,, X
[[," (V 1) X L. XX W,,2W,,2
(o Ln, to 1,1 X)X Lnn
[[o"" W W.2 oo
(W. X ( (W,,,,x X Lnn
(W Ln,s, + (W) Ln,4, XX WXWWeWWa
(W.) W In, (W, N.o.,./oWoW...K
Inn,o.j XW.22
+ x)
In,i + +o(W,,,xX)L, XJJ,
-
x LI +o(,,,1
+
XX W., o² (W,, Ln
+ =] W.,+ o ( x Ln.o.
( Ln,,x]ffc.X (X)
L.
+
Σ Mn.j00.0,ee exp((W...o(W..,xx)) x Mrn
1=1
+
+ (x) (x)
R,, (x)
(22)
where the coefficient Mn,..0,,0, == Ever, exp(bn.;) exp(ba.)) for any [7.].
22

---

Published as a conference paper at ICLR 2026
Step 2- Non-vanishing coefficients. An important insight from Equation 22 is that the ratio
Qn(2 ( can be expressed as a linear combination of the following independent functions:
E(X; W,,1,, WW,2 E(X; W.W... WW. x
[(X; W.,1j,
W.,, [ W W.,,j, W.
(W.,,,
W ,, B(X;
" B(X;
x W..2.
X
WV.
E(X; W. 17,, W. (W,,, x
X W E(X;
W. 22 (W ,, (4)
WW (() X ( R B(X; W. (W.,, Xo x) x X(1) X()) en:
[[KK: (())eu
X fa. (X), B(X;W,,,1 W. Sa, (X),
".X:W.,,,....... (W,,,)))
x( (X), E(X; W,, Ww,2)o (W.. X)x(1) x x (w)) fa,(X)
B(X; ,,,1,, (..))0 (W.1))
x [c., (X), F(X; W. 11,, W.2)o (W.,1, (W.,1) X) (u) x (0) sa, (X),
E(X; W W,,,, W. 22 (W 1,1 1, )o Jo X x x (w) fc. (X), ( W 2 fo. (X),
for any j S and S u,
d.
We now demonstrate that not all of the coefficients of these linearly independent functions converge to as n goes to infinity. To prove this by contradiction, assume that all these coefficients do converge to O as n goes to po. From the representation of the ratio Q ( in terms of these linearly independent functions, it follows that these ratios T, ['()) (h.i.//inn I, 2 /.2.//Din DDi / /Din / (e.5g//Dii ((wr) /. /'Dns, and Mn.j.0..o.//D all converge to as n approaches infinity for all indices u, U.E.E E [d] and j E [4].
By first considering the vanishing of the ratio 0y Din to O for any j [4] and then taking the absolute value of this ratio, we find that
MM...o..oal
Eiey, exp(br,)) - -xpp(aa)) - 0,
Din
Din
By varying the index j from to L and summing the corresponding limits, we find that
Σj EiEv, -o(0h.s)eex(lb. -
(23)
Din
Our strategy is now to consider separately the terms corresponding Voronoi cells with exactly one element and those with more than one element. In particular, for Voronoi cells with exactly one element, i.e., for indices j E [L] such that [Vj] = 1. as the ratio / /Dn approaches 0, we have
(u) Evey, exp(br.)) AlW,, 1811 Ed=1 L |
0.
Din
Din
Evev, exp(bn..) IWW. A,, ll
Similarly, the condition T,,22./'Di Din -> 0 implies that ing these results, we obtain
- 0. Combin-
Din
Enpy (xp((n,.)(I)AM) AW, u.l m.y + IlAW,, I11)
+ 0.
Din
Due to the equivalence of the l, and E, norms, we deduce that
EspA Evey, (xp(t..))lIAW,, 185 + AW,,.1
+ 0.
(24)
Din
23

---

Published as a conference paper at ICLR 2026
We now turn to Voronoi cells with more than one element, namely, indices As the ratios m Dn -> O, we find that
[L] such that [V]] 1.
Eu=1
(wu)
Eiev, exp(bn,))llAW,.. n.15!
nn,4J
0.
Din
Din
- O, we arrive Eiev, exp(bn,i) AW, _ 0. These results
Likewise, as (uu) 'D together imply that
Din
EADD Eiev, exp( ( (1 iigl n, 133)
0.
(25)
Din
Combining the results of Equations
23
22
and
25
we achieve that
D In
n oc,
Di
which is a contradiction. Consequently, at least one of the coefficients the linearly independent functions in the expression for the ratio (x) Dn does not converge to approaches infinity.
Step Application of Fatou's lemma. In this step, we divide all coefficients of the linearly (u) independent terms in the expression of the ratio (n(X)/DIn, namely, L("), L 2n,2.// /iin. [d]. by the
L /inn [ //.4.///i. Di, L /Din L /Din and Mn.j.0.,o.l Dn for all maximum of their absolute values. Specifically, we denote by m., the maximum of the absolute values of these coefficients. Since not all of these coefficients approach 0, it follows that 1/m,, does not approach infinity as n2 - OC.
From the hypothesis, have llfc. fa. (lg(()/ Din 0 as co. As 1/m,, A co, it follows that [| fc. (l Din, Din) + O. An application of Fatou's lemma yields
llfa. fc. (1(2(()
[cc. (X) fc. (X) dp(X).
lim 1I
lim 7=000
mn,
[fc. (X) fG. (X)
Combining these results, obtain lim inf simplify the presentation, we define the following limits:
0 for almost surely X. To
mn,D2n
Luns - ArJj
Mn..0a.0e
Ao.j=
m, Din
Din
for any 1 S < 6 and J j L. By the definition of mns at least one coefficient in the
sets {Aa. A1.s Aass Xs b..,,11111111 {No.j. >2.5 As,s, A As.s ber,, >>1 must be zeroz
[fc. (X) fc. (X) = O, or equivalently, lim inf mn
(n)
The condition lim i implies that
Din
(Wo XW,
+ +
XW o(W.,1, X)AI
Σ
exp((W..
X) XW.,2
+ +W +, XW., " +
+ ",
X W.,2
- A..XW.2
+ (,
T XW..
+o Ao.j + (W.,,, XX,
+ () X w Ao.5 +
XW.
(o( A5.j.
24

---

Published as a conference paper at ICLR 2026
(W,
X +W.., X) Sa, (X)
-
Σ
x)) (W.,, +oW,,
(W,, 1, X X XX, W o of Ao
+ (W W4, t (W,, Ao.5 fG, (X)
Aos (W).
Ao.s Exp((W (J =0
(26)
-=1
for almost surely X.
However, that equation implies that all the coefficients {Ao.j: A1.j )2j )3. (Ao.;
λz A3.j, As.j A-I, 0. It is contradiction to the hypothesis that at least one coefficient among these coefficients is different from
Consequently, we deduce that
lim e0+ GE) (G.G.)se (G..))))
inf
llfe fa. /o.llla
which proves the local part of inequality (20).
A.1.2 GLOBAL PART
From the local part of inequality (20) we can find positive constant e' such that the following inequality holds:
inf GEG (D): (G..
M/ Sc. /P 0.
To obtain the conclusion of the theorem, we only need to demonstrate that
inf GE (
M/ Sc. (L))) (G, G.) 0.
We prove the claim by contradiction. Assuming the claim does not hold implies that there exists sequence G", E exp(In.g 16 ...22) in the set GI/ (9) such that
W..2)
DI se {llee - fc. (. Dr - 0,
as n approaches the infinity. This implies that ||fe - fe. (le((1) + 0 as n approaches infinity.
By hypothesis, the parameter space θ is compact. Therefore, there exists a subsequence of G', 's, that converges to mixing measure G' where G' lies in the space (G)) From the hypothesis, we have D, (G', G.) > e'. By taking the limit both sides as n + oc, we obtain D (G',G.) G... 2 e'.
An application of Fatou's lemma leads to the following result:
lim |lfar - fa. 2 lim [|, (X) - fa. (x) dx()))d((()) --0
This inequality is only possible if Sar = Sc. for almost surely X.
By the identifiability of the function fo (x),, this equation only holds when G¹ G.. Consequently, Di (G', G.) = 0. This contradicts the assumption that Dz G.) 2 e' > 0. Hence, the proof of the global part is complete, thereby establishing the conclusion of the theorem.
Proof for the Identifiability Property. The key claim aim to show is that if the equation (x) = fa. (X) holds for almost surely X, then = G., namely, that the two mixing measures identical.
25

---

Published as a conference paper at ICLR 2026
From the hypothesis that fx(X) fc. (X) for almost all X, it follows that
exp(X T A/X a9))
i
exp((BW2o (W1g 1g X)) + by) (Wiy
11(,, .n,)) +
(x)
(x,
==1
j'=1
N exp(X X + d)
+
L
20 ()) + bes') 27 (X)
11X, n1))
+
(X)
(27)
where G = DL exp((1))60,,,, Furthermore, we define
Dj,o(() Σ exp(X AA2x +aa)
exp(BBW.2(W..IIx))) "X
X bn,g),
f'=1
N (x,,(X)= Σ exp(X
i
A
x + aQ) + Σ
(B by)).
This equation implies that the number of atoms of G and G, should be identical, namely, L. Therefore, the following equality holds
(Bp (W (W1xX)) x X + by)) : E 111 2xp((BW
X baoe) : (1)
(x,
Dj.G. (X)
for almost surely X. By relabelling the indices of these two sets, we can assume without loss of generality that
(W)( X)) by) exp((BW, 2,20 (W.,....11 X)) + ba.j))
(X,
Dj.G. (X)
for any index j' E [L] and for almost surely X. From the translation invariance property of the softmax function, the Equation (27 becomes
Σ exp (b,) (W,, (WIxX)
Σ
exp (b. (BW.((BW..
2W..,.
(28)
y=1
for almost surely X. This equation suggests that there exists a partition K₁, K2..... Km of the set [L] for some such that we have exp(b;,) exp(b=,jz) for any J1. K; and for any i [m].
Based on this result, the Equation (27) can be rewritten as follows:
exp (bj,) exp ((BW2 x)) (W1x))
jiek.
m
Σ exp (ba exp (BBW, XX)CW, X),
f=1 JeK,
for almost surely X. This equation proves that [W2a(W1,,XX) x): E K2} = (W.,20 (W.,1, X): :22 E Ki).
for any i (m.) Since the activation function a identifiable, this result indicates that {WW W2) : K,) = {(W.. W2) j2 K.).
Consequently, arrive the following equality:
ΣΣ exp (b,, (6,,), =
Σ Σ
exp (br.jz (6 .W..))
1= tiek
1=1 JoEK;
This is equivalent to G = G.. Thus, we obtain the conclusion of the identifiability property of the function fc.
26

---

Published as a conference paper at ICLR 2026
A.2 PROOF OF PROPOSITION
1
Recall that (X1, Vi), (X2, (,,, kn) E RR x Rdd follow the regression model:
Y fc. (X)) EG. == 1,2, ..........
where the independent Gaussian noises E,,..., En satisfy that B|s; [X,] - 0 and Var(s;|X.) - =²21x for all E [n]. The true regression function takes the following form:
N
exp X TTA
1 + ag)
SG. (X)
1x,, n1,
()
L exp((BW. (2) oW [.)) x)) X + ba.')) (w. [WW!)
Dx
where
:- (xx)==Ix exp X T a EE= exp(ex ( o(W!)
( X)) X +b..)) The
(
least-squares estimator G, is defined as
G arg wrraragmnnn minGeG
Σ Y - fo(X.)]l?
The assumption that E X ~ No L2 indicates that the least-squares estimator G, is indeed a maximum likelihood estimator, given by:
G E (0g
where p(Y p(Y.lff(X..) o² la is the multivariate Gaussian distribution with mean fa(X) and covari- ance matrix o² La. From the result of Theorem 7.4 in van de Geer (2000), it follows that
M(p(Y)a, oF) (ooYYa,..XX 0² Ix)) Op Op(Vlog(n)/r))
Given the closed-form expression for the Hellinger distance between two multivariate Gaussian distributions, we have
exp {
1 3/22 15a. (X) -x)3
802
Putting the above results together leads to
exep exp {
802 llfe, (X) - fo. (X)]1 (Oogln()))
Hence, for sufficiently large for some universal constant C' the above equality implies
1|/a, (X) - Ja. (x)11 < 80"
C Clo((n)/m/
So log
(I 1--I-Cloght I-Clog(n)/n)
C Clog(n)/n
Cllo((n//n
< S0²
1 1- C'lo(nn)/n
< 2602 Clog()lnn
This is equivalent to
llfa, (X) - (x)]l = 00ppV0o(p)v1)))
Consequently
llfa fc. (aat = (x) fo. (X)11((x(())x)
1/2
=Or( OOr(Voggn)//n)
The proof of the proposition completed.
27

---

Published as a conference paper at ICLR 2026
B ADDITIONAL THEORETICAL RESULTS FOR VISUAL ADAPTIVE PROMPT TUNING
Thus far, in Section 5 have provided theoretical benefits of visual adaptive prompt tuning when the function o in Equation (12) is nonlinear. In this appendix, we demonstrate that the visual adaptive prompt tuning also has appealing theoretical properties when the function or is linear.
Similar to the setting considered in Section we provide a theoretical guarantee for the linear setting of the visual adaptive prompt tuning via the regression framework. For empirical results, please refer to Appendix E.6
Problem Setup. We assume that the i.i.d samples of size (X1, Yi), (X2, 12), ..... (X.r Y,)
Rd Rdd are generated from the model:
Y fa_ (X,) t i= 1,2,..... ,
(29)
where &1, ......... En are independent Gaussian noise variables such that [E X 0 and Var(e; (X.)
2² Ta for all 1 sis Additionally, we assume that X1, X2.... X, are i.i.d. samples from some probability distribution j.. The regression function fā. (-) Equation (29) then takes the form
exp(X (
1 + +a9) 1(x,,5)) (X)
(x) = Σ
j=1
exp((BW.,2H) 2.W,,,,1W133 X)T x + +bby))
CW.,2W..117 W.
(30)
(x(X)
y=1l
with N pre-trained experts and L unknown experts, where 13 f. (X) = Ex)= exp (X 18
x a)) Σ exp( x) 1- +b.g)) while G. ES'=1 exp(ba.)' )orl 2......... denotes mixing measure, iee, weighted
of Dirac measures 8, associated with unknown parameters in the parameter space θ C R x dddd At the same time, the values of the matrix A9, the expert parameter ng. and the bias parameter ag are known for all 1 S S N. Additionally, the matrices B E dddd and C' E gdd xd are given.
(b..gr, W.
Least-Squares Estimator. In particular, consider the estimator
G = aringeee
Σ Y (V=x))(X.))
(31)
Voronoi Loss. The Voronoi loss tailored to the setting in Equation 30 is defined
D2. Σ
Σ EV
exp(b:) exp(ba.))
Σ
Σ
exp(tt)
VELLIiVV, k
Σ
Σ
where denote AWWWis bay W2W W. W.,2W.,17 for inj' Equipped with this loss function, we now provide the convergence rate of prompt estimation for the setting in Equation (30) in Theorem 2
Theorem Given the least-squares estimator G, defined Equation 31). we have that
D2(G,,, G_) O (((oog(n)/nn)
Proof of Theorem 2 Based on the convergence rate of regression function estimation presented in Proposition objective is to establish the following inequality:
inf (9)
||fc - (D, G.) > 0.
(32)
We partition the proof of the above inequality into local and global parts.
28

---

Published as a conference paper at ICLR 2026
B.0.1 LOCAL PART
The local part of the inequality
32 corresponds to the following condition:
[e - (lla()) 0. D2(G,G.)
lim
inf (O)...G))
We assume that the above inequality does not hold. Then, we can find sequence of measures Gn == Eÿ= exp(bn.g )ow, in ( such that as n + oo, we have
Dan = D2 G.) + 0, Ul fa. (allll D2 + 0.
Similar the proof of Theorem for clarity of presentation, V Vy denote the Voronoi cell of G. generated by the j-th component of the true measure G.. Since the ensuing arguments are asymptotic, without loss of generality we assume that these Voronoi cells do not depend on the sample size, i.e. we have V;- V for all and s j s L. Therefore, the Voronoi loss D2n can be rewritten as follows:
D2n
Σ TEV
exp(bn.i) exp(b
Σ
Σ
+
Σ
Σ
exp(bn.))
(WW. II?,
where we define AW,, 2W,,WWa,n Wiy == W,,WW,,1. n, W.,WW,,15 for all E Vy--
From the hypothesis that Din 0 as approaches oo, find that Eiev, exp(b..)) - exp(b..)) and WT Wn,2Wa,.1 > W. W 1, for any E Vi and j [T.]. Throughout this proof, we assume WLOG that B Ida. C = Tda and = for simplicity; we note that our techniques can be generalized to the general case. We partition the proof for the local part into three steps, as detailed below:
Step Taylor expansion.
We first define the following function:
N Qn(X): = Σ
exp(X
+
cxp((W..W..yx x+bns')) If.. (X) (X)
j=1
j'=l
Then, can decompose the function (x) as follows:
Qx(X)
exp(bn..) (xp((W 2wWn,WW..id cxp((WW..W..xx X)W.
exp (bp (xp(( X) (....W........X) x fc. (X)
j=1
exp(bn..) exp((b.,)) (,,.,,,XX/X///.WW..... fc. (x)
x) W.
-
1VV;
A, (X) (x) (X).
(33)
Decomposition of the function A, (X). To streamline the argument, we define the following functions: E W2 = exp(ex 2W2W X) x)Tx) X) and /l (X; W2W1) = W The product of the functions É and म defined as F(X; W2W,) = É(X; W Wi) To account for the differences in the number of components within each Voronoi cells, further decompose the
29

---

Published as a conference paper at ICLR 2026
function (X) as follows:
Ax
Σ
Σ exp(bn.i) Wn, ((.:
Σ
exp(bn.s)
F(x; F..
Am. (X) + (x)
An application of the first-order Taylor expansion to the functions E and A leads to:
gla E a(W,W;) (X:W.2W.,1)) + (x),
B(X;W,, = W +
(AWa,2
|0|=1
ala 1 + Σ 2w 1i)) a(W2W1a (x; W. W...W...... Rx,,(X),
Here, the indices j in these equations satisfy that [15] = 1, i.e. Voronoi cells with exactly one element and i E Vj. Furthermore, the functions Riy.1 and (x)((X) in these equations correspond to Taylor remainders when expanding the functions É and H. Putting the above results together leads to:
(x = Σ
exp(baa) ad!
alol (AW 8(W,W,)
alal
(AW,, n,2 W 1i) an.
(x:W. (W + R (X)
W(
gla E
Σ ΣΣ
Majaa
OWW,W))
JM -1 |a|=11
|a/=1
+ Mn..a DWW W.W.,
(X)
where (a1, 02). Furthermore, due to the uniform smoothness of the functions E and H, the remainder term R (X) from the preceding expansion satisfies (X))(())DA + when n approaches infinity. Finally, for any |a| = 1, the terms Mn,ja in the same expansion are given by:
exp(bn.i) (AW,.Wn..1)) ad!
Ma..o
REV,
We now turn to the decomposition of (x,) Unlike the analysis of (x,) which involved first-order Taylor expansion, to account for more than element in the Voronoi cells in (X)) resort the second-order Taylor expansions the functions E and Å. Specifically, applying second-order Taylor expansion yields:
(X).
alal E
Am,((X)=
Σ
Ma.ja )a
(X W. W.W.W.WW.... W.,, (X; W..WW...))
55
gl
Mnjaa W(
(X: E(X; W
gla E
Σ
M Mn.i...8 3 Ja W W.
a(2WW,) (X;W, + (X)
1a|=1, 1==
where ((1, 3 (.1,, 32). Due to the uniform smoothness of the functions È and i1, the term (x,, representng the combined Taylor remainders from this second-order expansion,
30

---

Published as a conference paper at ICLR 2026
satisfies Rx,2(X))25 - 0 goes to infinity. The coefficients Mn.j.a and Majja, in the expansion above are defined as follows:
exp(bn.)) a!!
Maj.a
iEV
for any coefficient a = (011 022) such that |a|| = 2 and
exp(bm.a)
Maja. = Σ
(AWn, Wn, 1s) a!!!!
1EV;
for any coefficients (a), 02) and B (31, .2) such that (a| = |8| 1. Given the definitions of
E(x; W1, W2) and (X;; W1, W2). their partial derivatives are computed as follows:
σ
X W2WW) = X (1a) (m))
0(W, W ((ure))
E-
W2W))) X x x (wz) x (1z)
exp((W2Wi1 X),
a(W 00 W1 (1222))
OH
(XWWWW, = (X (m
2(W ((rre))
X W2W1) Oa-
Putting the above formulations together, the functions 1 and 2x, can be rewritten as follows:
(x) = Σ
exp((W.pW..,,))
Σ
Mn
x (11) x (11)
M1
Mu.jeann x(vi)ems + 1X),
M1. M1.
(x,)(X)=
Σ
2 x)
Mnj.ca,, X x Wee
//V3/>1
My,0==1
+
Σ
Mo.ioco
(v1)em
Σ
Σ
M Mo.iccn toggo ( (11) X (w2) x (12) W..
=1.11=1
01,01=1 u2.02=1
Σ
Σ
M Majiceee C u000
(us) X (m)) X (12) Eug
Rx,
M1:P1
Ng.
Here, denotes the standard basis vector in Rd with in the u-th position and O in all other positions for any 1 < a < while Cuv denotes the matrix in daxd whose wi-th entry is and all other entries are O.
Decomposition the function B,(X), Similar to the decomposition of A, (X), we also separate the Voronoi cells with exactly element and with more than one element in the decomposition of the function B, (X). Therefore, obtain
Ba(X)= = Σ Σ
exp(bm.)) B(X -(;; (,,x:W,,W...13)) Sc., (X)
- iEV,,
Σ
Σ exp(bm..)) E(X; SX.XWW.,Wa..i. -B(
1,,1(() + (X).
31

---

Published as a conference paper at ICLR 2026
For terms corresponding to Voronoi cells with exactly component, first-order Taylor expansion. For cells with more than one component, we use second-order Taylor expansion. This strategy yields the following representations:
al E (X; W...., (X) + (X),
(x, = Σ
Ma.iac
1(W2W1)o
2/13
|a|=1
gla Mn.ja a(W2Wa)a (X; W W. (X) (1) (X).
(x) = Σ
In these expressions, the functions R, X) and R correspond the Taylor remainders. Due to the uniform smoothness of the function E, we obtain ( - and a + approaches infinity. By computing the closed-form expressions for the partial derivatives of the function É, both the functions B (X) and 2x, can be rewritten as follows:
(x)
E /
2 x)
Σ
Mn..es,n,
(( x((e))
faa (X) Rm,s(X)
M1,01=1
(x,((X)=
Σ
exp( W,,1 X)
Σ
M Madeor ((1) (
11,01=1
d
Σ
Mn.jean (ws) (v1) (w2) x (w2) fg. (X) + RX,(X),
01.21 =1 Mg,02=1
Plugging all of these results together, the function (X) can be rewritten as follows:
Qn(X)= 31
Σ
exp( x)
Σ
Mudieuj ( X(,
Σ
Musieure x x(m)e eus
01,01=1
01.01=1
d
Σ
[sp((W,.W.1557) x)
x)
Mu.ico ((u1) X(01) +
Masieuu X (v)
M1.01=1
01:01=
Mn.j.esiut tengry (w)) (v1) (uz) (x22 (
81 U1 u2
Mn.jeewgu Euguz X( (11) x (12) eua
01 M2
Σ
exp(
Xx)
M Museean x(11) x(11) fcr Fc. (X)
=
k1,t==1
Σ
x Tx)
Mn.icar (v:) (w1)
M1
Majeeuje ((m1 ((1)) ((2) x(02)) fo (X)
M1 #2.0n=1
L.
Σ Nns exp((WW,,W..,xx) "T)_:,_)xxxxxxxxx......... X)JG, (X)
1.
Σ
Nnj exp(
j=1
(x) + Rn,z(X) Rn, Rm,a(X
(34)
where Naj = Eiev, exp(bn.i) exp(ba.j) for any j [T]].
32

---

Published as a conference paper at ICLR 2026
Step Non-vanishing coefficients. An important insight from Equation 34 is that the ratio Qn (Piin can be expressed as a linear combination of the following independent functions:
B(X; W(X:W,,
X (1) W. IIX,
W. 2. ((1))
X (12) X (12) W., 1,,,, W2w.,) x (11) ("1) (12)) ewsr
[(X; [X:WW,,W...1 X (1)) X Sa.. (X), W(X:W. 1, X(()) ( X X (12) fa,
B[X; W,, (X); E W. W.,.W. W W. 0.2, X,
for any j S L, and for S u, v, w S d.
We proceed to demonstrate that not all of the coefficients of these linearly independent functions approach as goes to infinity. Assume, for the sake of contradiction, that all these coefficients approach 0 n goes to oo. From the representation of the ratio Qn (x)))/D2, in terms of these linearly independent functions, follows that the ratios Mn.j.a/ DDn Mn.ia.3/ aaaiaa.//2aaa DDn,, and Nn./ /ng/DDn approach 0 as oo, for all ax 8 E Ndxd such that 1 S |a| |3|
S
By first considering the condition Nn.j/De 0, it follows that Eiev, exp(bn.s) exp(ba.i)
[Nagl - 0,
D2n
D2n
for any J S L. By varying the index j from to L in these limits and summing them, we find that
Ej exp(bn.)) -xp((n.)exx(((.... exp(b<a)
- 0.
(35)
D2n
Our strategy now is to consider the limits corresponding to Voronoi cells with exactly one element and more than element separately. In particular, for Voronoi cells with exactly element, namely, for indices j E [L] such that their corresponding Voronoi cells have one element, i.e., [V1] = 1, as the ratio Mn.jeuu D2n >> O, we have
Σd, |Mn.dieu - 0. D2n Due to the equivalence between the & norm and the & norm, deduce that
Eiev, exp(bn..)) :I 2.
Dzn
E
+ 0.
(36)
D2n
We now move to the Voronoi cells with more than one element, namely, indices j E [4] such that |V > 1. Similarly, from the assumption that the ratio Mn.j,2e.../DD + O, we obtain
Evey,
Ed, Mn.j.2ewv 0.
(37)
Dzn
Dzn
Combining all the results in Equations
35
36 and 37
we obtain
D2n
+ 0, as n - oc,
Den
which is contradiction. As a consequence, at least one of the coefficients of the linearly independent functions in the expression of the ratio Q, (x) Drn, does not approach 0 as approaches infinity,
Step 3 Application of Fatou's lemma. The idea of this step is to divide all of the coefficients of the linearly independent terms in the expression for the ratio Q. (X) /D2.. namely, the terms Mn.j.a/ DDn. Mp.j...8 e/ DDn,, and Nn.; /Dn for all a, 8 E Naxd such that < |a| + [3] < 2, by the maximum of their absolute values. In particular, values of those coefficients. As not all of these coefficients go to follows that 1 //m, does not go to infinity XC.
first denote m, as the maximum of the absolute
33

---

Published as a conference paper at ICLR 2026
From the hypothesis, have |if.. - fä_ (al D2n 0 follows that llsG.. -Ga (m, O. An application of Fatou's lemma leads to
Since 1/m, A it
Sc. (X) -S. (x)
[r... Sā. ((Ea(H)
/
0 = lim
lim 1
du(x)) 20. 0.
my Mnn
myn 2m
fc. (X) - fa.. (X)
Combining these results, obtain lim inf +00
0 for almost surely X. To
mnn D2n
simplify the presentation, we define the following limits:
Mn.ja Aj.n m, 2n
Mnj.a. mn
Nu.j Tj
mn D2n
for any 1 < < I and a E Naxd such that < la| + 11] I 2. By the definition of mn-
least one coefficient in the set {) Sn.0. Tj : j aL). E (Nxdd" 1 S la + |8| 2]
fa. (X) fā. (X)
must be non-zero. Then, the equation lim in
0, or equivalently,
m, D2n
(x) leads to
lim inf 400
my D2n
2xp((W.2W.,11j x)
Σ
X ((m))(
Σ
Ajennss X (e))ea
Cus
////==1
11,01=
Σ
exp((W.,W.25xx x)
Aglenyy x(11) Wo
Aa.aum x(1) Ear
///11>1
01,
01
Aoooyyy tGggg ((1)) (((1))((())))) (12)) (11)) WW,,
01,01= =1 u2
((1)) (01) X x(02) Euz
Sh Guguu
11:P1
Airunyy x (ui) x(er) (X) (X)
=
Σ
x)
Aauujyy X(a1 ((11)
3//1111
Eieayey Bugoz
((11) X (1z) x(e))
Sa. (X)
01.00
1.
Tj CC
+ Σ Tj Cxpp(( 2 3=1
(38)
for almost surely X. However, the new equation implies that all the coefficients (Ax E, Ty j E a, 3 E Ndxd 1 5 Jal + | < 2} are O, which is contradiction.
As consequence, we obtain
inf (O) which proves the conclusion of the local part of the inequality 33
lim +
llfe - fa. /DG 0,
34

---

Published as a conference paper at ICLR 2026
B.0.2 GLOBAL PART
From the result of the local part of the inequality (32). we can find a positive constant e' such that the following inequality holds:
inf ( (O): G,e To obtain the conclusion of the theorem, we only need to demonstrate that
Ilfa - fä. / (G) 0.
inf GEg, (G):D.
llfa - fa. /l
0.
We prove the claim by contradiction. Indeed, by assuming the claim does not hold implies that there exists a sequence of G', = Σラ= exp(bm.5:) J6w... in the set (O) such that
D2(G",, G.) |lfec - fll (2(G) G.) - 0,
as long as n approaches infinity. This implies that |fa - (121l + as n goes to infinity.
From the hypothesis, the parameter space θ is compact. Therefore, a subsequence of G", 's converges some mixing measure G' where G' lies in the space (z)( From the hypothesis, have D2 D2(G,, G.) e'. By taking the limit both sides as n + oc, we obtain D2(G", G.) e'.
An application of the Fatou's 's lemma leads to the following result:
(X) 0 0= lim ||f - fa. (cal) / lim fox (X) - fa. (x) du(x).
This incquality only possible fa = fa. for almost surely X.
According to the identifiability of the function fc(X), this equation only holds when G' =²= G.. As
consequence, we obtain D2'G', G.) = O. This contradicts the deduction that (G", G.) (.))2=' 2 > 0. Hence, the proof of the global part is completed. This achieves the conclusion of the theorem.
Proof for the identifiability property. The key claim that we aim to show is that the equation fx(X) = fa. (X) for almost every X, then G = Gz, namely, the two mixing measures are identical.
From the hypothesis, fo(X) fo. (X) for almost all X, we have
cxp(X
X T X + ag)) 11(X,1) .1/)
i
exp((BW2W, 13 X)T X + bb))
Σ
CW2W15
(x)
(x.
y==1
N exp(X' 19 + aQ))
exp((BW.,. (BW.BBBWWW,.. 2,2W,,15 ..X. x+bb")))
h1 1(Xx,1) n1))
CW CW.2WW,,1 X,
D D (X)
D (X)
(39)
where G Σi exp(b)))ww, Furthermore, we define
N
(X) (x)a Σ exp(X x A + a
1.a..
exp((BW..2W.11 X) + b,,g),
k=1
=l
D Σ exp X
10x + ag)
exp((BW2W1,)) x) by).
f'=1
That equation implies that the number of atoms of G and G. must be identical, namely, we have = f.. Therefore, the following result holds
X) by) : E [41]
exp(( (BW
W...2 T bay') ; 111).
(x,
D D,,0, (X)
35

---

Published as a conference paper at ICLR 2026
for almost surely X. By relabeling the indices of these two sets, assume without loss of generality that
e X + bby) exp((BW,,,W) x) b..i')
(x,(()
D Dy.3 (X)
for any index 1' E [L] and for almost surely X. From the invariance to translation property of the softmax function, the Equation 39 becomes
exp (0,))xx(((B))) exp(( X) X)
exp (b, (xp())exxBBBB. X)CWW,2W..1p (X)C,x
(40)
for almost surely X. This equality suggests that there exists a partition K1, K2,..., Km of the [L] for some m such that we have exp(b)) = exp(b=jz) for any j1,J2 E K; and for any i E [m].
According to this result, Equation (40) can be rewritten follows:
exp (6)) exp (BB2W11, X)C
1= JIEK,
exp (b (BWW..ex..B....,ppp XX)Cxx X)CW.
f2EK,
for almost surely X. This equality implies that
(W2W1,, X : ji E K2} = 2W. X
x: f2 E Ki).
for any i [m]] This result indicates that
(W2W1n : K.) = (W.2WWWp. :12 K.).
As consequence, we arrive at the following result:
exp (bj, Jowsw,,
Σ
exp (br,jz 2
JIEK;
1=1 JsEK;
This is equivalent to G... Consequently, the identifiability property of fa established.
C RELATED WORK
Mixture of Experts. Building classical mixture models with adaptive gating mechanisms (Jacobs) et al., 1991 Jordan Jacobs) 1994, Xu et al.) 1994). the MoE model has been extensively studied and refined over the years. Notably, subsequent works (Eigen al.) 2014 Shazeer al., (017) introduced the MoE layer as an efficient tool for scaling model capacity. Unlike traditional models
that apply uniform parameters all inputs, the MoE layer applies specific parameter subsets for each
input, creating a sparsely activated layer that enables significant capacity growth without proportional increase in computational cost (Fedus et al., 2022, Zhou al. al.], (003). This efficiency and scalability
have driven its adoption across diverse domains and tasks (Riquelme et al. 2021 Du et al. 2022
Shen et al. 2023).
Theory of Mixture of Experts. Although MoEs have been widely used to scale up large models, their theoretical foundations remain under active development. For example, Ho et al, (2022) focused on input-free gating Gaussian MoEs and showed that under maximum likelihood estimation, the experts' convergence rates depend on the algebraic independence of the expert functions. Next, Nguyen al. [2023 2024a) established convergence rates for both density and parameter estimation in Softmax coftmax-gating Gaussian MoEs, linking these rates to the solvability polynomial systems under Voronoi-hased loss functions. More recently, Nguyen al. (2024blc) employed least-squares
36

---

Published as a conference paper at ICLR 2026
estimation identify conditions under which expert functions are identifiable. Under these conditions, the resulting estimation rates improve substantially.
Parameter-Efficieer Fine- Tuning, Fine-tuning pre-trained foundational models (Dosovitskiy 2020
Liu et al., 2022 Kirillov et al. 2023) has become a widely adopted strategy for tackling downstream tasks (Xin et al., 2024). While effective, full fine-tuning is rexource-intensiive, requiring updates to all network parameters and the storage of a separate fine-tuned model for each task (Han et al.) 2024). To address these challenges, researchers have increasingly focused on parameter-efficient fine-tuning (PEFT) methods. These methods can be categorized into partial tuning, extra module, and prompt tuning. Partial tuning freezes most of the backbone, fine-tuning only subset, such as linear heads or a few layers (Mahajan et al. 2018 Chen et al.) [02T] He et al., (002). Extra module methods add trainable parameters to the backbone, such as side structures (Zang et al., 2020), residual MLP modules in Transformer layers (Houlsby et al., 2019; Cai et 20020), or low-rank weight updates (Hu et al. 202T). Despite their promise, partial tuning and extra module methods often face limitations that restrict their applicability. First, they may fail to achieve performance comparable to full fine-tuning (Mahajan et al. 2018 Chen et al. 202 I Jia et al.] 2022 He et al.
2023). Second, some methods rely on specific architectural modifications (Rebuffi et al., 2017 Cai
et al. 2020 Zhang et al.. 22200), limiting their generalizability across different backbone architectures.
Prompting Methods. Prompt tuning (Lester et al. 2021 ) initially proposed for language tasks, offers a simpler yet effective alternative by introducing learnable parameters to the input sequence of backbone models, updating only these parameters during fine-tuning (Lester et al., [02T]] Jia itt al. 2022 Le et al., 2025bla). ay. Despite its simplicity, prompt tuning has demonstrated significant performance gains. Recently, visual prompt tuning has emerged as a promising paradigm in PEFT techniques for computer vision. Current advancements in visual prompt tuning focus on engineering improvements, such as minimizing parameter usage (Han et al., 2023) and broadening applicability to diverse tasks (Yao et al., [2223] Sohn et al., 2023; Yao et al., (024). However, the theoretical foun- dations of prompt-based methods remain underexplored. For instance, He et al. (202T) investigated the relationship between prompt tuning and adapter methods, while|Le et al. (2024) analyzed these
techniques within the context of MoE models. Furthermore, [Petrov et al. (2023) highlighted the limitations of prompting, showing that it cannot alter relative attention patterns and instead biases attention layer outputs in a fixed direction.
Adaptive Prompting. A growing body of work investigates adaptive prompting strategies for visual prompt tuning (Zhou et al., (202). Several approaches employ meta-learning or clustering techniques to tailor prompts specific downstream datasets. In particular, HHuang et al. (2023) and Kim et al. (202)) cluster the downstream data and assign cluster-specific prompts based on the ViT's initial input representation, ((), Similarly, recent prompt-pool methods (Wang et al. 2022 Kim et al. 2024) first forward the input image through an encoder network to obtain [CLS] representation, which is then used to select or generate prompts at every attention layer. In all these cases, the prompt
experts effectively become functions of the first-layer input, X (0) rather than of the current layer input, (1)
Our proposed method, VAPT, instead generates prompts at each layer conditioned on the layer-specifer representation, X (1) This layer-wise adaptation follows directly from our theoretical framework, which establishes a connection between MoE and VPT and implies that both the experts and their gating functions should depend on (²) This perspective enables more principled and analytically grounded examination of model behavior. Importantly, the motivation for VAPT stems from the functional role of prompt experts and their distinction from pre-trained experts, rather than from purely empirical desire to assign different prompts to different inputs.
Moreover, while some adaptive prompting approaches rely on heuristics complex image-to-prome transformations that make their effects difficult to interpret (Huang et al.) [2024; Xiao et al. 2025). VAPT adopts a simple yet effective formulation. This design choice leads to the clean expert definitions in Equations (13) and (14. and supports a rigorous theoretical analysis without sacrificing empirical performance an aspect largely absent prior work.
Most recently, Zeng et al.) (2025) explicitly introduce MoE components into the Transformer archi- tecture, including new routing modules for sparse prompt activation. contrast, our analysis in Section 3 shows that an implicit MoE structure is already embedded in existing visual prompt tuning frameworks. Consequently, VAPT does not require additional architectural components or routing
37

---

Published as a conference paper at ICLR 2026
mechanisms. Instead, it refines the formulation of prompt experts within this inherent implicit MoE structure. essence, VAPT enhances the expressive capacity of the underlying prompt experts while leaving the backbone architecture unchanged.
D IMPEEMENTATION DETAILS
Caltech101
CIFAR-100
DTD
VTAB VTAB-Nature Natural
8
680
EurOSAT
Reslac45
Patch Camelyon
VTAB -TAB-Specialized
Retinopathy
Clevricount
Clevriistance
KITTicistacce
VTAB Structured
dSpettestarkentation
SmatNORBlooluuth
SmalHORBleevvtion
eSpriteslocationn
FGVC
CUB- 200 2011
Stanford Dogs
Figure 3: Dataset examples for all classification tasks evaluated
Dataset Specifications. Table provides summary of the statistics and details of the classification datasets evaluated in this paper. presents image examples from all 24 datasets. Following Dia al. 2022), each FGVC dataset is randomly split into 90% train in and 1 10% va 1, with the va 1 set
38

---

Published as a conference paper at ICLR 2026
Table Specifications of the datasets used in our experiments, which include 24 datasets from two benchmarks: FGVC and "TAB-IK (Zhai et al. 2019).
Dataset
Description
# Classes
Train
Validation
Test
Fine-grained visual recognition lasks (FGVC)
CUB-200-2011 (Wab
2011
Fine-grained secognition bird species
200
5.794
NABirds
Van Morn et di
2013
Fine grained recegnition of bird species
55
21,536
2,393
24,633
Oxford Flowers Nillbbck Zisserann
Fine-grained secognition flower species
102
6.149
Stanford Dugs (Khosla et ai 2011
Fine grained recegnition of dog species
120
10,800
1,200
8,580
Stanford Cars (Gebru
sine-grained socognition
8,041
Visual Task Adaptation Benchmark (VTAB-IK)
CIFAR-100 Krizhevsky et al..200 2009
100
10,000
Caltech101 Pei-
102
6,084
DTD Ci 201-
47
1,880
Howers (Nlabbac Zisserman
Natural
800/1000
6,149
Pots (Parkh 2rar 2017
37
3,669
Netzer
Sun397 XIB el mr. 2010
397
21,750
Patch Camelyon (Veeling
2018
2
32,768
BuroSAT (Helber 0 2Uy
Specialized
10
800/1000 200
5,400
Resise45
C 207
6,300
Retinopathy Uriham
2013
5
42,670
Clevricuunt Johnssnn et 2017
8
15,000
Clevridistance
6
15,000
DMLab ef 2010
6
22,735
KITTIdrsance (Ge)p
2013
Structured
4
800/1000
711
dSprites/loc (Mattey et Ai.
2017
16
73,728
dSprites/ori (Mainney
16
73,728
SmalINORB/azi LXC
2004
18
12,150
SmallNORB/cle
used for hyperparameter tuning. For VTAB-IK, an 800-200 split for tuning, and we train all available data for the final evaluation.
Pre-trained Backbone Specifications. This study explores two primary groups of Vision Trans-
former (ViT) models. The first group comprises conventional VIT architectures (Dosovitskiy 2020).
including ViT-Base. ViT-Large, and ViT-Huge, mre-trained the large-scale ImageNet-2IK (Deng
ct al. 2009) dataset. The second group focuses on self-supervised models, specifically Masked Autoencoders (MAE) (He et al., 2022) and Momentum Contrast v3 (MoCo v3) (Chen al. 2021), both pre-trained on ImageNet- 1 K. Unless otherwise specified, we use ViT-B/16 with supervised pre-training on ImageNet-2IK dataset by default.
Augmentation. During training, we employ standard image augmentation techniques. For the five FGVC datasets, images are normalized using ImageNet mean and standard deviation, followed by random resized cropping to 224 x 224 and random horizontal flipping. For VTAB- 1k, images are directly resized 224 224.
Hyperparameters. We use the val 1 set of each dataset to find the optimal prompt length Np, kernel size K, and hidden dimension of the feature projector. Following Dia et al. (002), the search range for Np is {1. 5, 10, 50, 100, 200}. Given the relatively small resolution of our feature map Xing (e.g.. 14 14 for ViT), the kernel size K is selected from (2, 3, 4). For the hidden dimension r, we explore the range {4, 8, 16, 32, 64, 128, 256). To identify the optimal learning rate and weight decay, we perform a grid search, consistent with Mahajan et al.) al.) (218); Jia et al. (2022). The learning rate searched within {50, 25, 10, 5, 2.5, 1, 0.5, 0.25, 0.1, 0.05). while the weight decay is chosen from (0.01, 0.001, 0.0001, 0.0}. We adopt the batch size settings of Ji et al. (2022), using values of 64 and 128. The models are optimized with SGD for 100 epochs, employing cosine decay learning rate schedule with 10 warm-up epochs.
Reproducibility. Our method, VAPT, is implemented in PyTorch Paszke al. 2019). All experi-
mental workflows, including training and evaluation, were conducted on NVIDIA A A100-40GB GPUs. The entire implementation will be made openly available ensure reproducibility and facilitate future research.
39

---

Published as a conference paper at ICLR 2026
Table VTAB-IK Natural per-task results for ViT-B/16 supervised pre-trained on ImageNet- 21K. "Number of Wins" in [-] denotes the number of wins relative to full fine-tuning. "Tuned/Tota represents the percentage of parameters tuned for each task. The highest accuracy among all approaches except Full is highlighted in bold.
VIT-B/16 (Desvviskly, 2020
VTAB- IK DЛTD>
Zhai et al. 2019
Natural [7] Pers
(83.500
CIFAR-100
CaltechIDI
SVHN
Sun.997
Mean
Full
Tofinova al. 2022
68.9
87.7
64.3
97.2
86.9
87.4
38.8
75.88
Linear
Teativ RE 7022
63.4
85.0
63.2
97.0
86.3
36.6
51.0
68.93 [1]
[2]
MLP-2
CTen df 2420
63.2
84.8
60.5
97.6
85.9
34.1
47.8
67.20 [2]
MLP-3
Chen
63.8
84.7
62.3
97.4
84.7
32.5
49.2
67.80 [2]
[1]
MLP-9
Clea
A120
53.1
80.5
33.9
95.1
82.6
24.4
43.7
61.90 [1]
Sidetune /Zhans
2020
60.8
53.6
95.5
Bias (Rebali 2.
201
72.8
87.0
59.2
97.5
85.3
$9.9
51,4
73.30 [3]
Adapter-250 [C 2
2020
74,1
$6.1
63.2
97.7
87.0
34.6
50,8
70.50 [4]
Adapter.64
85.8
62.7
87.2
70.65
Adapter-8 ca 2020
74,2
85.7
62.7
97.8
87.2
50.7
70.67 [4]
VPT-Shallow Jia
2022
77.7
86,9
62.6
97.5
87.3
74.5
51.2
76.81 [4]
Tuned/
VPT-Dcep Uia all
2023
78.8
90.8
65.88
98.0
88.3
78.1
49.6
78:48 [6]
Tuned7 Tocal
0.20
0.20
0.15
0.10
0.04
0.54
0,41
0.23
R2VPT (Han
80.01 [6]
Tanis tocal (5)
0.22
0.19
0,11
0.11
0.05
0.24
0.43
0.19
VAPT
69.7
98.8
(0.05)
(0:42)
[6]
Tunee// Toral (%)
0.15
0.16
0.11
0.13
0.07
0,42
0.18
Table 6: VTAB-1K Specialized per-task results for ViT-B/16 supervised pre-trained ImageNet- 21K. "Number of Wins" in [-] denotes the number of wins relative to full fine-tuning. "Tuned/Total" represents the percentage of parameters tuned for each task. The highest accuracy among all approaches except Full is highlighted bold.
ViT-B/16 (Dosovisskiy. 2020
VTAB-IK (Zhai et EuroSAT
2019] Specialized [4] Resise45
Mean
(85.8M)
Patch Camelyon
Retinopathy
Full (Tofinova ct al.
2022
79.7
95.7
84.2
73.9
83.36
Linear [ohnova et 2022
78.5
87.5
68.6
74.0
77.16 [1]
Partial-
Yosınski al. 2014
78.6
89.8
72.5
73.3
78.53 [0]
MLP-2
Chen ct al.
2020
74.3
88.8
67.1
73.2
75.86 [0]
MLP-3
Chen et al.
2020
77.0
88.0
70.2
56.1
72.83 [0]
MLP-5
Chen et al.
2020
73.7
87.2
64.8
71.5
74.31 [0]
MLP-9
Chen
2020
78.5
83.0
60.2
72.3
73.49 [0]
Sidetunc (Zhang
2020
58.5
87.7
65.2
61.0
68.12 [0]
Bias (Rebuth et al. 2017
78.7
91.6
72.9
69.8
78.25 (0)
Adapicr-256 (Cai ct al.
2020
76.3
88.0
73.1
70.5
76.98 [0]
Adapter-64 Cai et 2020
76.3
87.5
73.7
70.9
77.10 [0]
Adapter-8 Cai al.
2020
76.9
89.2
73.5
71.6
77.80 [0]
VPT-Shallow (Jia
al.
2022
78.2
92.0
75.6
72.9
79.66 [0]
Tuned / Total (%)
0.01
0.05
0.09
0.01
0.04
VPT-Deep [Jia et al.
2022
81.8
96.1
83.4
68.4
82.43 [2]
7 Total
1.06
1.07
0.15
0.02
0.57
E2VPT (Han al. 2023
82.5
96.8
84.8
73.6
84.43 [3]
-Tuned Total
(%)
0.20
0.29
0.12
0.07
0.17
VAPT (Ours)
84.4 + (0.72)
96.5 +: (0.09)
85.1 + (0.46)
74,5 + (0.32)
85.13 [4]
-Tuned / Total (%)
0.30
0.35
0.09
0.06
0.20
E ADDITIONAL EXPERIMENTS
E. PER-TASK RESULTS FOR VTAB-IK AND FGVC
Table Table 6 Table 7 and Table g provide per-task results across classification tasks evaluated in Table I All results averaged of three using different initialization seeds, with standard
40

---

Published as a conference paper at ICLR 2026
Table 7: VTAB-IK Structured per-task results for ViT-B/16 supervised pre-trained on ImageNet- 21K. "Number of Wins" in [-] denotes the number of wins relative to full fine-tuning. "Tuned represents the percentage of parameters tuned for each task. The highest accuracy among all approaches except Full is highlighted in bold.
VIT-M/16 (Dosvviskiy 2020
VTAB-IK K Zhni et al. 2019) Strictied [S]
(85.SM)
Clevz!
Cleve! distance
DMLLab
KITTD distanoz
aSyeites location
dSgrites/ orientation
SmallNORB/
SmalINORB/ elevation
Meaa
azimuth
Full Tofinme 30255
56.3
Partial-11
2014
41.5
34.3
61.0
31.3
16.3
22.4
MLP-2
Chtc 00
2000
45.2
31.6
31.8
55.7
30.9
24.5
16.0
23.3
32.47 (0)
MI.P-3
Lib 1
20137
47.8
32.8
12.3
58.
12.9
21.2
15.2
24.8
90.62 (0]
MLP-5
LiB
X1.1
50.8
12.3
31.5
S6.4
7,5
20.8
14.4
20,4
29.25 [0]
MI.-9
47.5
27.9
28.9
$4.0
62
17.7
10.5
16.2
36.15 [0]
7smret
27.6
R
Adagter444
42.9
39,9
3044
54.5
31,9
25.6
13.5
21.4
32.51 (0)
Adipter-8 Car
(060)
45.2
41.8
31.1
$6.4
30.4
24,6
13.2
22.0
$3.09 [0]
UPT-Shallow
di 2022
50.5
$8.6
40.5
67.1
68.7
36.1
20.2
34,1
46.98 (4)
Tuned/
0.10
0.09
0.19
0.13
VPT-Deep al. br23
68.5
72.8
32.9
Tuned7
2.12
elanet 3023
71.7
61.2
47.9
75.8
48.1
31.7
41.9
$7.39
Turgu7 19at 1o.r
0.34
0.65
0,44
0.26
0.10
0.38
1.14
0.06
0.51
VAPI
74.8 1: (1.70)
63.6 d: (0.36)
50.0 1: (0.74)
77.2 (0.72)
86.1 1: (0.24)
48.3
(0.89)
33.8 (0.95)
40.9 d: (2.29)
59.34 [8]
TTane// T Total
0.20
0.39
0.51
0.38
0.08
0.35
0.60
0.75
0.38
Table 8: FGVC per-task results for ViT-B/16 supervised pre-trained on ImageNet-2IK. "Number of Wins" in [-] denotes the number of wins relative to full fine-tuning. "Tuned/ notee//otaal" represents the percentage of parameters tuned for each task. The highest accuracy among all approaches except Full highlighted in bold.
VIT-B/16 (Dssovisskiy 2020
FGVC [5] Oxford Flowers
Mean
(83.8M)
CUB-200-2011
NAbirds
Stanford Dugs
Stanford
Full (lofinova et al 2022
87.3
82.7
98.8
89.4
84.5
88.
Linear
(Totnova el al.
2022
85.3
75.9
97.9
86.2
51.3
79.32 [0]
Partial-
Yosiinki
2014
85.6
98.2
85.5
66.2
82.63
MLP-2
Chen
85.7
77.2
98.2
85,4
54.9
80.28
MLP-3
Chen al
2020
85.1
77.3
97.9
84.9
53.8
79.80
MLP-5
C Chen ct al
2020
84.2
76.7
97.6
84.8
50.2
78.71 [0]
MIP-9
Chen cl
20:20
83.2
76.0
96.2
83.7
47.6
77.31 [0]
Sidetune (Zhang et al._ 2020
84.7
75.8
96.9
85.8
48.6
78.35 [0]
Bias (Rebuth et al. 2017)
88.4
84.2
98.8
91.2
79.4
88.41 [3]
Adapicr-256 Car al.
2020
87.2
98.5
89.9
68.6
85.70 [2]
Adapter-64
2020
87.1
98.5
89.8
68.6
85.67
Adapter-8
87.3
84.3
98.4
88.8
68.4
85.46
VPT-Shallow Jia
2022
86.7
98.4
90.7
68.7
Tuned/ Totn
0.31
0.54
0.23
0.20
0.26
0.31
VPT-Deep Jia al.
2022
88.5
84.2
99.0
90.2
83.6
89.11
Tuned 1o
0.29
1.02
0.14
1.17
2.27
0.98
E2VPT (Han et al. 2023
89.1
84.6
99.1
90.5
82,8
89.22 [4]
Tunod Total
0.32
0.65
0.15
0.88
1.27
0.65
VAPT (Ours)
89.7 + (0.12)
84,6 +: (0.06)
99.1 + (0.04)
91.7 + (0.05)
82.8 + (0.53)
89.58 (4)
Tuned Total (%)
0.36
0.79
0.19
0.53
2.04
0.78
deviation error bars included. Compared VPT and other commonly used PEFT methods, VAPT demonstrates consistently superior performance across a variety of downstream tasks while utilizing fewer parameters.
E.2 PER-TASK RESULTS ON MAE AND MoCo v3
Tables 10 and provide per-task results for MAE (He al. 2022), as summarized in Table 2 and [4 presentt per-task results for MoCo v3 (Chen et al. (202I). which are
Similarly, Tables 12
also summarized Table 2
41

---

Published as a conference paper at ICLR 2026
Table VTAB-IK Natural per-task results for ViT-B/16 pre-trained on MAE (He al., 2022).
"Number of Wins" in [-] denotes comparisons relative to full fine-tuning. "Tuned/Total is the per- centage of parameters tuned for each task. The highest accuracy among approaches is highlighted
in bold.
ViT-B/16 Dosoviiskiy 2020
VTAB- IK Caltech101
(Zhai et al. (2119)
Natural [7]
Mean
(85.8M)
CIFAR-100
DTD
Flowers T02
Pets
SVHN Sun397
Full (lofinova et al. 2022
24.6
84.2
56.9
72.7
74.4
86.6
15.8
59.31
(APT()
34.4
89.7
63.1
74.2
73.8
55.1
24.3
59.23 [5]
Tuned/ / Total (%)
0.17
0.16
0.10
0.13
0.07
0.12
0.40
0.16
Table 10: VTAB-IK Specialized Per-Task Results for ViT-B/16 Pre-trained on MAE (He al., 2022). "Number of Wins" in [-] denotes comparisons relative to full fine-tuning. "Tuned/Total" is the percentage of parameters tuned for each task. The highest accuracy among all approaches is highlighted in bold.
ViT-B/16 Dosovitskiy (85.8M)
2020
VTAB-IK (Zhai al.
2019)
Specialized [4] Retinopathy
Mean
Patch Camelyon EuroSAl Resise45
Full [ofinova al. 2022
81.8
94.0
72.3
70.6
79.68
(Our)
78.9
91.6
78.7
73.7
80.73 [2]
-Tuned / Total (%)
0.36
0.31
0.15
0.03
0.21
Table 11: VTAB-1K Structured Per-Task Results for ViT-B/16 Pre-trained on MAE (He et al., 2022). "Number of Wins" in [-] denotes comparisons relative to full fine-tuning. "Tuned/Total" is the percentage of parameters tuned for each task. The highest accuracy among all approaches is highlighted in bold.
ViT-Base/16 Desovitskiy. 2020
VTAB-IK Zhai et al. dSprites location
2019 Structued (8)
(85.8M)
Cleve/ count
Cleve! distance
DMLab
KITTH
dSprites! orientation
SmalINORB/
SmalINORB/ elevation
Mean
distance
azimuth
Full Jotinova et al. 2022
67.0
59.8
45.2
75,3
72.5
47.5
30.2
33.0
53.82
VAPT (Ours)
57.9
57.1
37.3
68.2
82.6
11.5
41.7
47.24 (2)
Tuned/ / Total (%)
0.18
0.65
0.33
0.35
0.09
0.36
0.66
0.67
0.41
Table 12: VTAB-1K Natural Per-Task Results for ViT-B/16 Pre-trained on MoCo v3 (Chen et al. 2021). "Number of Wins" in [-] denotes comparisons relative to full fine-tuning. "Tuned/Total is percentage of parameters tuned for each task. The highest accuracy among all approaches highlighted in bold.
ViT-B/16 Dosovitskiy 2020 (85.8M)
VTAB-IK (Zhai et al. 2009)) Natural [7]
Mean
CIFAR-100
Caltech101 DTD 100errs122
Pets SVHN Sun397
Full [Tofiooaa et al. 2022
57.6
91.0
64.6
91.6
79.9
89.8
29.1
71.95
(Our)
74.5
92.0
69.5
93.2
88.2
84.6
41.8
77.69 [6]
Tuned / Total (%)
0.15
0.20
0.09
0.34
0.09
0.10
0.40
0.20
E.3 STATISTICAL SIGNIFICANCE TESTS
To determine whether VAPT consistently surpasses competing PEFT methods across the 19 VTAB tasks, we conducted one-tailed paired Wilcoxon signed-rank test [Wilcoxon] 1992) for each VAPT- baseline comparison. The null hypothesis Ho for each comparison The alternative hypothesis H performance scores between VAPT and the baseline method is zero. was that VAPT performs better than the baseline method. As reported in Table 15 all resulting p-
that the median difference in
42

---

Published as a conference paper at ICLR 2026
Table 13: VTAB-IK Specialized Per-Task Results for VIT-B/16 Pre-trained on MoCo v3 (Chen et al. 2021). "Number of Wins" in [-] denotes comparisons relative to full fine-tuning, "Tuned/ the percentage of parameters tuned for each task. The highest accuracy among all approaches is highlighted in bold.
ViT-B/16 Dosovitskiy 2020 (85.8M)
VTAB- IK (Zhai al. Patch Camelyon EuroSAl Resise45
2019
Specialized [4] Retinopathy
Mean
Full [Iofinoaa al. 2022
85.1
96.4
83.1
74.2
84.72
VAPT
80.6
95.9
83.9
75.4
83.95 [2]
-Tuned / Total (%)
0.36
0.34
0.11
0.03
0.21
Table 14: VTAB-IK Structured Per-Tase Task Results for ViT-B/16 Pre-trained on MoCo v3 (Chen al. 2021). "Number of Wins" [-] denotes comparisons relative to full fine-tuning. "Tuned/Total" the percentage of parameters tuned for each task. The highest accuracy among all approaches is highlighted in bold.
ViT-Base/16 Dosovitskiy 2020
VTAB-IK Zhai et al. 2019] Structired [8] SmalINORB/ azimuth
(85.8M)
DMLab
KITTI!
dxprites location
dSprites! orientation
SmalINORR/ elevation
count
distance
distance
Full Jotinova et al. 2022
55.2
56.9
44.6
77.9
63.8
49.0
31.5
36.9
51.98
VAPT (Ours)
88.0
60.74 (7]
Tuned Total (%)
0.20
0.69
0.05
0.36
0.08
0.20
0.66
0.65
0.36
values were below the 0.05 significance level, indicating that the observed performance improvements of VAPT are statistically significant.
E.4 DIFFERENT BACKBONE SCALES
94
VAPT
92
90.6
90
89.7
Accuracy
88
87.3
86
84
82
80
VIT-Base
MT-Large
VIT-Huge
Figure 4: Comparison VAPT and VPT on CUB-200-2011 across different backbone scales.
In Figure 4 we compare VAPT and VPT on the CUB-200-2011 dataset (Wah et al. 2011 using pre- trained backbones of varying scales (ViT-Base, ViT-Large, and ViT-Huge). The results demonstrate that VAPT consistently surpasses VPT the model size increases. Notably, with the ViT-Huge back- bone, VAPT achieves up to a 1.3% improvement over VPT. These findings highlight the scalability and effectiveness of VAPT compared VPT as model size grows.
43

---

Published as a conference paper at ICLR 2026
Table 15: Wilcoxon signed-rank test evaluating whether VAPT significantly outperforms other methods across 19 VTAB tasks. The results show that VAPT is statistically significantly better than other baselines (p 0.05).
Method
Full
Bias
Adapter
VPT
E2VPT
p-value
5.7e-06
1.9e-06
1.9c-06
1.9c-06
0.0001
Table 16: Semantic segmentation results ADE20K. All methods evaluated with SETR (Zheng et al. 2021) using ViT--.. The best mloU scores among all methods but Full are bolded.
Method
Full
Head
Bias
VPT
VAPT
mloU
48.31
35.12
43.40
42.24
44.04
params
318.31
13.18
13.46
15.60
15.29
E.5 SEMANTIC SEGMENTATION
To explore the potential generalizability of VAPT beyond visual classification, evaluated its performance on the semantic segmentation task. We report mloU values in Table[T6] Our observations indicate that VAPT attains a higher mloU than VPT while introducing fewer additional parameters. Furthermore, VAPT remains competitive with other PEFT approaches. These findings highlight VAPT's strong generalizability across various computer vision tasks.
E.6 ABLATION STUDY
Table 17: Impact of Different Components in VAPT. Experiments were conducted on the VTAB- 1K benchmark. We used 6 Supervised Pre-trained ImageNet-21K the backbone. "Tuned/Total" is the percentage of parameters tuned for each task. Bold highlights the best results.
Components
Tuned Total
VTAB-IK IK-IK (Zai al.
2019
Mean Total
Channel-wise
Feature projector
Sharing Projector
Natural
Specialized
Strucfured
81.43
85.13
59.34
72.91
80.47
84.63
58.13
71.94
0.26
79.43
83.60
57.72
71.17
0.42
80.85
85.10
59.04
72.56
Impact of Different Components. We conducted ablation studies on the VTAB- IK benchmark (Zhai ct al. 2019) to evaluate the individual contributions of each component in VAPT. The results, presented in Table 17 show that incorporating the channel-wise convolution layer inereases the overall average performance by 0.97%. This improvement highlights the benefit of explicitly encoding spatial relationships in the feature map before it is processed by the token-wisg siien-wwieeee projectors. Furthermore, the channel-wise convolution layer reduces the overall parameter count by downsampling the feature map from H W to H' W", where H' Hl K + and W" W K Consequently, the dimensionality for each token- wise projector is also reduced, leading to fewer parameters. When the feature projector removed, performance decreases; for instance, in VTAB-IK Natural, the performance drops from 81 .43%% to 79.43%. This decrease highlights the importance of the feature projector. Furthermore, our sharing mechanism for the feature projector not only reduces the number of parameters but also facilitates knowledge transfer between layers, leading to improved performance. Overall, combining all components yielded the best performance, with an average accuracy of 72.91% % on VTAB-1K. K..
Detailed Analysis the Channel-wise Convolution Layer. We evaluated the effectiveness of channel-wise convolution layer by comparing its performance to that of standard convolution layer. We also explored alternative strategies for modeling spatial relationships, including an average pooling layer (LeCun et al. 1998). Notably, average pooling be considered special case of
44

---

Published as a conference paper at ICLR 2026
Table 18: Detailed Analysis of Channel-wise Convolution. Experiments were conducted on the VTAB-IK benchmark. We used ViT-B/16 Supervised Pre-trained on ImageNet-21K as the backbone. "Tuned/Total" is the percentage of parameters tuned for each task. Bold highlights the best results.
Tuned Totall(%
VTAB-IK (Zhai et al.
2019
Method
Mean Total
Natural
Specialized
Structured
Channel-wise Convolution
0.27
81.43
85.13
59.34
72.91 72.20
Standard Convolution Average Pooling
0.33
80.79
84.81
58.37
0.27
81.30
85.02
58.41
72.45
our channel-wise convolution layer, where the kernel weights are fixed rather than learned. The comparative results are presented in Table Our channel-wise convolution layer not only reduces the number of parameters compared to a standard convolution layer but also mitigates overfitting, thereby improving overall performance. Furthermore, it surpasses the average pooling layer; for instance, on VTAB-IK Structured, we achieve 0.93% performance gain. This improvement is attributed the greater flexibility our channel- -wie convolution layer compared that of average pooling.
(a) VTAB-IK Sun397
(b) VTAB-IK Retinopathy
(c) VTAB-IK Clevr/distance
Figure 5: Ablation on prompt length Np. Results are reported datasets, each corresponding to distinct VTAB subgroup. The dashed line indicates the best results of VPT.
(a) VTAB-IK Sun397
VTAB Retinopathy
(c) VTAB-IK Clevr/distance
Figure 6: Ablation on kernel size K. Results are reported on 3 datasets, each corresponding to distinct VTAB subgroup. The dashed line indicates the best results of VPT.
Robustness to Different Hyperparameters. We systematically evaluated the influence of key hyperparameters, including prompt length Np. kernel size K, and hidden dimension r. by conducting experiments three representative VTAB-IK tasks: Sun397 (Natural), Retinopathy (Specialized), and Clevr/distance (Structured). The results are shown in Figures and As depicted, the optimal hyperparameters vary across tasks. Nevertheless, VAPT consistently achieves higher performance than VPT across a range of these hyperparameters. Notably, even with only single prompt, VAPT maintains its advantage over VPT.
Linear Activation in the Feature Projector. As shown in Appendix B VAPT preserves its optimal sample efficiency when the activation function in the feature projector (see Equation (11) is
45

---

Published as a conference paper at ICLR 2026
114
(a) VTAB-IK Sun397
(b) VTAB-IK Retinopathy
(c) VTAB-IK Clevr/distance
Figure 7: Ablation on hidden dimension r. Results are reported on 3 datasets, each corresponding to distinct VTAB subgroup. The dashed line indicates the best results of VPT.
Table 19: Linear activation in the feature projector. Experiments were conducted on the VTAB- IK benchmark. We used ViT-B/16 Supervised Pre-trained on ImageNet-21K as the backbone. "Tuned/Total" is the percentage of parameters tuned for each task. Bold highlights the best results.
Tuned T'otal(%) (%)
VTAB-IK (Zhai et al. Specialized
2019
Method
Natural
Structured
Mcan Total
VPT
0.69
78.48
82.43
54.98
69.43
VAPT linear
0.27
80.89
84.93
59.28
72.64
VAPT non -lin-r
0.27
81.43
85.13
59.34
72.91
replaced by linear identity function. To substantiate this theoretical result, we report the correspond- ing performance in Table 19 where a is removed. The results demonstrate that the linear version of VAPT remains competitive with the non-linear variant. Notably, it still outperforms VPT by substantial margin (e.g., 4.30% on the VTAB- IK Structured task), confirming both the theoretical and empirical robustness of our approach.
E.7 COMPUTATIONAL CoST
Table 20: Comparison of FLOPs and MAC' for VAPT and VPT. The experiments conducted on FGVC benchmark. We used ViT-B/16 Supervised Pre-trained on ImageNet-21K as the backhone.
Metric
Method
Stanford Cars
CUB-20I-2011
Oxford Flowers
NABirds
Stanford Dogs
VAPT
73.67 (f 0.18%)
37.04 (f 0.11%)
36.10 (f 0.08%)
44.61 0.31%)
54,30 (f 0.59%)
FLOPs (GFLOPS)
VPT
73.54
37.00
36.07
44.47
53.98
VAPT
36.80 (f 0.16%)
18.51 (f 0.11%)
18.03 (1 0.06%)
22.29 (1 0.32%)
27.13 (1 0.59%
MACs (GMACs)
VPT
36.74
18.49
18.02
22.22
26.97
One primary concern when designing prompts that adapt to the input is the associated computational overhead. Although VAPT outperforms VPT, one notable advantage VPT its simplicity, as its prompts remain fixed regardless of the input. To examine these trade-offs, we compare both the performance and computational costs of the two methods. Table 20bbortt these costs, measured in FLOPs (GFLOPs) and MACs (GMACs) for VAPT and VPT across five FGVC datasets. VAPT incurs only a slight increase in computational cost, with FLOPs increasing by 0.18% for Stanford Cars and ranging between 0.06% and 0.59% across other datasets. A similar pattern is observed for MACs. These modest increases are outweighed by the significant performance gains. Notably, VAPT also employs fewer parameters than VPT. Overall, these findings highlight the efficiency of VAPT, delivering robust performance improvements at a minimal additional computational cost.
E.8 ADVERSARIAL ROBUSTNESS
To evaluate the adversarial robustness of VAPT, we apply the Projected Gradient Descent (PGD) attack (Madry al. 2017 VTAB-IK CIFAR-100 test samples. As illustrated in Figure g model
46

---

Published as a conference paper at ICLR 2026
17.33
Accurary (%)
16
13.37
11.87
PGD Steps
Figure 8: Comparison of VAPT and VPT performance on VTAB-IK CIFAR-100 under PGD adversarial attack.
Table 21: Image-Caption Retrieval results ImageNet. We compare VAPT against VPT using CLIP ViT-B/32 backbone in 16-shot setting. The models are evaluated using Recall @ 100 on the validation set. Bold highlights the best results.
Recall @ 100 (%)
Method
Image
Text
Text -> Image
Average
VPT
79.05
74.80
76.93
VAPT (Ours)
82.66
77.72
80.19
Improvement
+3.61
+2.92
+3.26
performance degrades significantly under attack, which is expected given the limited training data and the absence of adversarial-speciff training methods (e.g., adversarial fine-tuning). Despite this, VAPT consistently outperforms VPT under adversarial conditions, suggesting that it may confer some robustness benefits. However, these findings are preliminary, and further experiments are required to substantiate the observed trends.
E.9 INTERPRETIVE VISUALIZATIONS
To facilitate deeper understanding of our method, we provide visualizations to illustrate the advantages of VAPT. Specifically, we use GradCAM (Selvaraju et al.] (017) to generate attention maps by computing the gradients of a target concept with respect the model's final layer. Figure] 9 presents examples from five VTAB-IK IK datasets, namely Sun397, SVHN, Resise45, Clevr/count, and KITTI/distance, comparing heatmaps generated by VAPT and VPT. These visualizations enable us examine the regions of the input data to which each technique directs the model's focus. We observe that VAPT can localize relevant image regions more accurately than VPT. For instance, the Sun397 dataset, while VPT struggles to capture the complete structure of object, VAPT in identifying and highlighting its key features. This enhanced localization indicates VAPT's stronger ability capture salient visual patterns. Consequently, VAPT only improves performance VPT but also enhances the interpretability of the model by providing more coherent and precise visual explanations.
succeeds
E.10 MULTI-MODAL EXPERIMENTS
To investigate the generalizability of Visual Adaptive Prompt Tuning beyond uni-modal visual recog- nition, we extend our evaluation to a multi-modal setting, specifically image-caption retrieval. This experiment assesses whether the enhanced functional expressiveness of adaptive prompt experts facilitate better alignment between visual representations and textual semantics within a contrastive learning framework.
47

---

Published as a conference paper at ICLR 2026
Input
VPT
VAPT
Input
VPT
VAPT
Sun397
SVHN
Resisc45
Clevr/count
KITTI
Figure 9: GradCAM visualization of VPT and VAPT five VTAB-IK datasets. Red regions indicate areas of higher class activation. From left to right: the input image after standard data augmentation, the GradCAM output from VPT, and the GradCAM output from VAPT.
Experimental Setup. We perform image-text retrieval the ImageNet dataset, enriched with rich natural language descriptions as proposed by Fang et al. (2022). We employ the pre-trained CLIP ViT-B/32 model (Radford et al. 2021) the backbone architecture. Consistent with main experiments, VAPT is applied exclusively to the visual encoder, while the text encoder remains frozen. We adopt few-shot learning protocol, utilizing 16 samples per class, which amounts 16,000 total training samples across the 1,000 ImageNet classes. Both the baseline VPT and our VAPT method utilize a prompt length of Np 10, resulting to the backbone. The models are optimized using SGD with learning rate of 0.01 and global batch size of 64. The training duration is set to 2 epochs, utilizing the first epoch for warmup, and the objective function is the standard symmetric cross-entropy loss used in CLIP pre-training. Evaluation conducted on 10,000 samples from the ImageNet validation set, reporting Recall 100 (R@ 100) for both Image-to- (I+T) and Text-to-lmage (T-+l) retrieval directions.
marginal parameter increase of only 0.19% relative
Results. The empirical results, summarized in Table 21 demonstrate that VAPT significantly outperforms VPT in the few-shot cross-modal retrieval task. VAPT achieves average R@ 100 of 80.19%, surpassing the VPT baseline by 3.26% %. This performance gain is consistent across both retrieval directions, with improvements of 3.61 % for Image-to-Tent and 2.92% for Text-to- Image retrieval. These findings suggest that the input-conditional nature VAPT prompt experts allows the visual encoder dynamically emphasize features that correlate more effectively with natural language descriptions. Consequently, VAPT achieves superior alignment in the multi-modal embedding space compared to static prompting strategies, without necessitating extensive parameter updates or modification of the text encoder.
F
USE OF LARGE LANGUAGE MODELS
Large language models were employed solely for editorial purposes, including grammar correction and spelling refinement. They were not used for content generation, data analysis, or the design of experiments.
48


Reviews:
[{"review_id": "9h1Ae5sPQE", "reviewer": "Reviewer_DQQA", "summary": "This paper reinterprets Visual Prompt Tuning (VPT) through the lens of Mixture of Experts (MoE) and identifies a key limitation: conventional VPT uses static, input-invariant prompt tokens, which restricts expressiveness. Building on this observation, the authors propose Visual Adaptive Prompt Tuning (VAPT)—a parameter-efficient extension that generates input-dependent prompts. The authors empirically demonstrate consistent gains over VPT and other PEFT methods across VTAB-1K and FGVC benchmarks.", "questions": "- Can the proposed visual adaptive prompt method be applied on top of recent VPT variants?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The main motivation of this paper can provide a mathematically grounded analysis\n\n- The paper is easy to follow.\n\n- The authors provide a variety of experiments, with results on FGVC, VTAB-1K, and supervised and self-supervised pretrained backbones, showing the robustness of the proposed method. In addition, ablation studies in the Appendix are very helpful to understand the proposed method.", "weaknesses": "- My major concern is the novelty.\nFor adaptive visual prompt tuning, there are many visual prompt tuning works (e.g., CVPT, CoCoOp, ViaPT, V2APT) already exploring visually adaptive or instance-aware prompts. Hence, the contribution in adaptivity itself is incremental rather than fundamentally new. In addition, MoE Interpretation is also heavily motivated by Le et al, who already framed attention and prompting under MoE theory. \n\n\n[CVPT] CVPT: Cross-Attention help Visual Prompt Tuning adapt visual task, NeurIPS 2024\n\n[CoCoOp] Conditional Prompt Learning for Vision-Language Models, CVPR 2022\n\n[ViaPT] Visual Instance-aware Prompt Tuning, MM 2025\n\n[V2APT] Visual Variational Autoencoder Prompt Tuning, arXiv 2025\n\n\n- The paper lacks sufficient comparison with other recent variants of Visual Prompt Tuning. There has been a surge of follow-up works for VPT, yet these are not adequately discussed or compared. Especially, input-dependent or adaptive prompting methods should be carefully compared and discussed.\n\n- There are some fairness issues, where the Tuned/Total ratio differs substantially across methods. For instance, Gated VPT (in Table 2) uses only about 0.05 % of trainable parameters, while VAPT tunes 0.27 % – 0.28 % of the model. Such discrepancy can partly explain the performance gap, making the comparison less fair. A more rigorous evaluation should control for the number of trainable parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lw7qMjBOmU", "reviewer": "Reviewer_YSjb", "summary": "This paper revisits Visual Prompt Tuning through the lens of Mixture of Experts and identifies a fundamental limitation: conventional prompt tokens are static and input-invariant, thus lacking expressive power. To address this, the authors propose Visual Adaptive Prompt Tuning, which generates input-dependent adaptive prompt experts using lightweight token-wise projectors, channel-wise convolutions, and a shared feature projector. This design enhances the functional expressiveness of prompts while maintaining parameter efficiency. Theoretically, VAPT achieves optimal sample efficiency under the MoE framework, and empirically, it outperforms both fully fine-tuned and VPT baselines across VTAB-1K and FGVC benchmarks.", "questions": "Please address questions in Weakness section. Thank you.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper offers a clear theoretical reinterpretation of VPT through the lens of MoE, providing both conceptual insight and mathematical grounding for understanding prompt tuning behavior.\n\n2. The proposed Visual Adaptive Prompt Tuning (VAPT) effectively enhances the expressiveness of prompt experts by introducing input-dependent adaptive prompts while maintaining parameter efficiency.\n\n3. Overall writing is clear and easy to follow,", "weaknesses": "1. Conceptually, VAPT’s “input-adaptive prompt experts” is similar to prompt-pool-based approaches [R1. R2]. These methods also condition prompt selection or generation on input features. Especially, [R2] generates tokens based on visual prompts based on the input. If authors could provide comparison between proposed method and existing  prompt-pool-based approaches, it strengthens the novelty of  works.\n\n[R1] Wang, Zifeng, et al. \"Learning to prompt for continual learning.\" CVPR 2022.\n\n[R2] Kim, Youngeun, et al. \"Open-world dynamic prompt and continual visual representation learning.\" ECCV 2024.\n\n2. The proposed approach introduces multiple small components (channel-wise conv, token-wise projector, shared MLP). While lightweight in current ViT-B/16 settings, their scalability to larger backbones (ViT-L/14, ViT-H/14) or higher-resolution inputs is not discussed. The added modules could potentially become computational bottlenecks or require additional tuning.\n\n3. Although the paper argues that VAPT enhances functional expressiveness, there are no visual analyses (e.g., attention maps, learned prompt diversity, or feature attribution) to substantiate this claim. Qualitative results could help illustrate how adaptive prompts differ in behavior from static ones.\n\n4. (optional) All experiments are performed on classification and segmentation benchmarks. It would strengthen the contribution to show that VAPT generalizes to non-classification visual tasks, such as detection or vision-language retrieval, especially since prompt-tuning is often used in multimodal settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HreZGdesyP", "reviewer": "Reviewer_sjhP", "summary": "The paper reinterprets VPT as the introduction of new prompt experts into these MoE structures, solving the current limitation in existing VPT frameworks: the restricted functional expressiveness of prompt experts.", "questions": "The major concern is the novelty and some claims in this paper. The most relevant paper on MoE prompt tuning is not discussed in this paper. Also, the paper sounds like the direct integration of MoE. Even without a similar approach, the novelty itself is questionable. The authors do not clearly separate their method from traditional MoE attempts.\n\nAnother problem is that some claims might be misleading, though the authors formulate the MoE and the proposed method's training objective; the core idea is intuitive and simple. I think some equations are unnecessary and further complex the understanding of the basic concept of this paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The experiment settings are sound with sufficient numbers of ablation studies in the Appendix. \n2. The paper is easy to follow, and the motivation for introducing MoE to prompt tuning is reasonable.", "weaknesses": "1. The baselines provided in this paper are not new. More recent prompt tuning and other PEFT methods [1-4] should be included for completeness. \n\n2. A critical problem of this paper is its novelty; [5] has proposed MoE prompt tuning as a manifold mapper, indicating that MoE design on prompt tuning can bring stronger expressivity. This work is highly related to the proposed research, although it has not been discussed. \n\n3. Inconsistent experimental report. Table 1 includes E2VPT, while Table 2 does not. Similar for GateVPT.\n\n4. The claim on the low-data regime in the introduction does not have further showcases. Also, the noticeable performance gap might be an outline for VPT. The reason is, as shown in [6], VPT generally brings good few-shot performance, when the training repeats for several times (to avoid bad samples for the training). \n\n[1] Visual Fourier Prompt Tuning\n\n[2] Visual instance-aware prompt tuning\n\n[3] Apla: A simple adaptation method for vision transformers\n\n[4] DS2VP: Dynamically-Selected Spatially Visual Prompting\n\n[5] MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper\n\n[6] Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)