Visual Prompt-Agnostic Evolution
Authors: Junze Wang
Lei Fan
Dezheng Zhang
Weipeng Jing
Donglin Di
Yang Song
Sidong Liu
Cong Cong
Venue: ICLR 2026 Poster (2026)
Keywords: Computer Vision
Visual Prompt Tuning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Visual Prompt Tuning (VPT) enables effective adaptation of a frozen Vision Transformer (ViT) to downstream tasks by inserting a small number of learnable prompt tokens into the token sequence at each layer. However, we observe that existing VPT variants often suffer from unstable training dynamics, characterized by gradient oscillations. A closer layer-wise analysis reveals that shallow-layer prompts tend to stagnate early, while deeper-layer prompts exhibit high-variance oscillations, leading to a cross-layer mismatch. These issues contribute to slower convergence and degraded final performance. To address these challenges, we propose the Prompt-Agnostic Evolution ($\mathtt{PAE}$) method, which can strengthen vision prompt tuning by explicitly modeling the dynamics of learnable prompts. From a frequency-domain perspective, we initialize prompts in a task-aware direction by uncovering and propagating frequency shortcut patterns that the backbone inherently exploits for recognition. To ensure coherent evolution across layers, we further employ a shared Koopman operator, which imposes a global linear transformation rather than uncoordinated, layer-specific updates. Finally, inspired by Lyapunov stability theory, we introduce a regularizer that constrains error amplification during evolution. Extensive experiments demonstrate that using $\mathtt{PAE}$ with VPT variants not only accelerates convergence with an average 1.41$\times$ speedup but also yields 1–3% gains on 25 datasets with multi downstream tasks. Beyond performance, $\mathtt{PAE}$ remains prompt-agnostic and lightweight, and it integrates seamlessly with diverse VPT variants without backbone modification or inference-time changes, providing a practical and scalable solution for advancing prompt tuning.

Published as a conference paper at ICLR 2026
VISUAL PROMPT-AGNOSTIE EVOLUTION
Junze Wang"r, Lei Fan"; Dezheng Zhang', Weipeng Jing', Donglin Di", Yang Song", Sidong Liu", Cong Cong
University of Science and Technology Beijing, UUniversity of New South Wales Northeast Forestry University, TTsinghua University. "Macquarie University wangjjuzzeyeehh. net, lei. Fan1@unse edu. au, thomas congêmg. edu, au
ABSTRACT
Visual Prompt Tuning (VPT) enables effective adaptation of frozen Vision Transformer (ViT) to downstream tasks by inserting small number of learnable prompt tokens into the token sequence at each layer. However, we observe that existing VPT variants often suffer from unstable training dynamics, characterized by gradient oscillations. A closer layer- wise analysis reveals that shallow-layer prompts tend to stagnate early, while deeper-layer prompts exhibit high-variance oscillations, leading to cross-layer mismatch. These issues contribute to slower convergence and degraded final performance. To address these challenges, we propose the Prompt-Agnostic Evolution (PAE) method, which strengthen vi- sion prompt tuning by explicitly modeling the dynamics of learnable prompts. From frequency-domain perspective, we initialize prompts in task-aware direc- tion by uncovering and propagating frequency shortcut patterns that the backbone inherently exploits for recognition. To ensure coherent evolution across layers, we further employ a shared Koopman operator, which imposes a global linear trans- formation rather than uncoordinated, layer-specific updates. Finally, inspired by Lyapunov stability theory. we introduce a regularizer that constrains eπтоr ampli- fication during evolution. Extensive experiments demonstrate that using PAE with VPT variants not only accelerates convergence with an average 1.41 x speedup but also yields 1-3% gains on 25 datasets with multi downstream tasks. Beyond per- formance, PAE remains prompt-agnostic and lightweight, and it integrates seam- lessly with diverse VPT variants without backbone modification or inference-ting changes, providing practical and scalable solution for advancing prompt tuning")
INTRODUCTION
With the advent of large-scale vision foundation models, particularly those based on Vision Trans- formers (ViT') (Dosovitskiy et al., 2020), significant progress has been made in learning general- purpose visual representations that
be transferred to wide range of downstream tasks (Zhu ct al. 2025; Fan et al. 2025a by Cong et al. 2025). However, effectively adapting these powerful pretrained models to specific tasks remains key challenge, especially under constraints such as limited labeled data and computational resources. To address this, prompt tuning has emerged as lightweight and parameter-efficiet solution, enabling task-aware adaptation without modifying
the backbone network (He et al. 2023 Mai et al., 2025). ). Among these approaches, Visual Prompt Tuning (VPT) (Jia al. (022)) has shown particular promise for ViTs. By introducing small set of learnable prompt tokens, VPT enables effective adaptation to new tasks while preserving gener- alizable knowledge encoded in the pretrained backbone (Xiao et al. (025c).
Subsequent VPT variants improve upon this along four main directions: Structured prompts (Jia
al. 2022 Han et al., 2023 Tu et al. 2023] Xu et al. 2025; Ren al. 2025 Wang al. (025b). which enrich input with additional channels, e.g., E2VPT (Han et al. (2023) adds key/value prompts
and prunes tokens for efficiency, and ProVP (Xu et al. (025) reorganizes prior-layer outputs and ap-
plies contrastive learning to reduce mismatch; Adaptive prompting mechanisms (Wang et al. 2022
"qqual contribution.
1 'Corresponding author.
'The code is available at athnps:/ggihub..co/rrec/////

---

Published as a conference paper at ICLR 2026
Accurncy & Converpence Speed
Layer-Mean Gradients
Shallow ve Deep Gradient Dynamics
Awampn Graaber L2-Noom
Avergu Gradbezz L2 Nomi
1of
50
Top-1 Accuney (16)
10"
Spock 22
Toue To
Bock
V91
Daey (V97)
82N77
Skadion (VP7)
Ency (1P) 2340
Spicl 1t
5S5VP
10
Banr (VPT
Skikon (VP7- EMA
191 PMU
Shal on (VP7 PAID
20
40
60
50
40
50 100
50
Epuch
Fppoch
Epuic
(a)
(b)
(c)
Figure 1: (a) Comparison of accuracy and convergence speed is shown in multiple VPT variants, including VPT, VPT+EMA, E2VPT, VFPT, and SA2VP. (b) Gradient oscillation (12 layers mean) is observed in multiple VPT variants, Lee, VPT, E2VPT, VFPT, and SA2VP. (c) VPT hierarchically exhibits shallow-layer (Layers 1 1-4) stagmation and deep-layer (Layers 9-12) oscillations.
Dong et al.
2023 Sohn et al. 2023 where methods like LPT (Dong et al. 2023 dynamically com-
bine shared and group-specifie prompts for long-tailed tasks; Projection-bused prompts (Xiao al.)
2025a b Liu et al., (225) tightly couple input projection with prompt optimization, and Perception- driven designs (Xu et al., 2024 Zeng et al.) 2024; Wang et al., 2024; Zhou et al. 2024). such as VFPT (Zeng et al., 2024). which reweights spectral components in the Fourter domain. Intuitively, since VPT and its variants update only small number of prompt parameters, one might expect faster and more efficient training. In practice, however, they often suffer from slower convergence and sub- optimal accuracy (Figure Th). To understand this discrepancy, we analyze their gradient behaviors during training. Figure [b) reveals that many VPT variants exhibit pronounced gradient oscilla- tions, particularly in the early and middle training stages. Further, our layer wiie-wise&ee gradient analysis in Figure TT) shows a clear mismatch in optimization dynamics between shallow and deep layers. Specifically, prompts inserted into shallow layers experience an early surge in gradient magnitude followed by stagnation, remaining close their initialization, while deeper-layer prompts begin to oscillate heavily after this stagnation sets in. These phenomena suggest poor coordination across layers and contribute directly to the observed degradation in training efficiency and performance.
We attribute these phenomena to two persistent challenges in VPT variants. (1) Task- aanostg Prompt Initialization. While prior works have explored various initialization strategies for prompt tokens (Jia et al. 2022] Wang et al., 2024] Xu et al.] 2025; Kang et al.) 2025), these initializations are generally agnostic to the downstream task. As a result, early gradient updates tend to align the prompts with the pretrained backbone rather than the target task (Wang et al. (025b). To expe- dite this alignment phase, many methods employ aggressive optimization setting with high initial learning rates which can exacerbate instability and induce gradient oscillations, ultimately hinder- ing convergence and degrading performance. (2) Independent prompt design across layers. In most VPT variants, prompts pre-pended and optimized independently at each transformer layer. During training, only representations propagate forward through the model, while gradients must pack-propagate through many frozen layers. This leads to significantly weakened gradient signals in the shallow layers, causing their prompts to stagnate near initialization values (Merlin et al.) 2023 Meanwhile, deeper-layer prompts are repeatedly adjusted in compensation, resulting in am- plified gradient oscillations. The lack of explicit cross-layer coordination introduces an optimization mismatch across layers, contributing to slower convergence and reduced final accuracy.
To address Challenge (I), we propose Modal Pre-Alignment (MPA), inspired by recent findings that pretrained vision backbones often rely on specific frequency shortcuts to make correct prediction (Wang et al.) 2025a). MPA performs a lightweight search to identify these task-aware shortcuts and uses them to initialize the visual prompts. To address challenge (2), we reformulate prompt optimization as a Koopman (Mezić) L (Shevitz& & Paden] (194)) discrete dynamical system (KLD). this formulation, the prompt embedding at each layer is treated as the system state, and its evolution is governed by single shared operator that maps the prompt state from one layer to the next. Rather than optimizing each layer's prompts independently, this perspective explicitly couples them through common dynamical rule, thereby establishing cross-layer dependencies.
Building on these two components, we introduce Prompt-Agnostic Evolution (PAE), a unified method designed to stabilize and accelerate prompt learning for VPT variants. PAE operates in two sequential stages: First, MPA provides task- prompt initialization by leveraging frequency- domain cues aligned with the backbone's 's inductive biases. Then, KLD evolves these prompts across
2

---

Published as a conference paper at ICLR 2026
layers using shared Koopman operator, enforcing smooth inter-layer transitions. A Lyapunov-style regularizer further enhances stability by constraining error accumulation during evolution. Notably, PAE is lightweight, introduces inference-tine overhead, and can be seamlessly integrated into existing methods without modifying the backbone. Our contributions are summarized as follows:
To the best of our knowledge, this is the first work to reframe VPT as the control of prompt trajectories within a dynamical system. This offers an explicit cross-layer association per- spective for VPT variants.
We propose MPA, a aware prompt initialization which utilizes Siquenncy-dmmalle cues produce prompts that aligned with the downstream objective from the start.
We propose KLD which employs a Koopman evolution operator to establish cross-layer association of prompts, while ensuring the stability of the entire optimization process by Lyapunov-style regularizer.
Extensive experiments on 25 datasets establish new state-of-the-art performance by in- corporating PAE into various VPT variants, yielding faster convergence without incurring additional inference-time costs.
METHOD
2.1 PRELIMINARIES AND PROBLEM FORMULATION
Formulation. Given a dataset DD consisting of input-label pairs (I, y). where x E R x3 is an image and y E (1,...,C } is the corresponding class label, the image is processed by Vision Transformer (ViT) backbone F. composed of L encoder blocks {E Specifically, æ is first passed through a patch embedding layer. which splits into N patch tokens. As shown in Fig- (r) learnable visual prompt P; E R T is pre-pended to the patch tokens at each transformer layer i, where T denotes the prompt length and is the embedding dimension. We denote the set of prompts across all layers as P [PI.... P... These prompts are optimized while keeping the VIT backbone I' frozen. The final classification head is trained jointly with P and the overall training objective is defined as:
min E( FI ((F(; y).
P
where Lusk is a task-aware loss function. The goal of PAE is to accelerate and stabilize the learn- ing of prompts P by combining two complementary components: MPA, which provides task-aware initialization, and KLD, which enforces coordinated optimization across layers.
2.2 INITIALIZATION VIA MODAL PRE-ALIGNMENT (MPA)
To initialize (Pinit) V= with task awareness, we propose the MPA strategy. As illustrated in Fig- ure De),,, MPA consists of two stages, First, we identify frequency shortcuts by probing F using the training dataset D. Next, we use the identified shortcuts construct the first-layer prompt Pinit which is then propagated through F to generate the complete initialization set (Pinin Ja=2.
Phase I - Discovering Frequency Shortcuts. Neural networks often exploit the most distinctive frequency components in data as shortcuts during training (Wang et al. (023). However, such fre- quency biases are not explicitly represented in the spatial domain, making them difficult observe or control directly. To address this, we transform images into the spectral domain, where frequency characteristics are more accessible for analysis. Specifically, we begin by sampling a mini-batch {Zzj, w;)} from the training set. Each image z, is transformed into the frequency domain using the 2D Fourier transform, yielding F(z,) If To identify task awaee regions in the frequency spectrum, we define set of binary masks [,,] ) each corresponding
local region of the fre- quency space. These masks are generated by sliding a window of size w X w with stride along both height and width, resulting S = (1 H- + +1) (1 W- + 1) distinct window locations. Fach mask M, {0, 1) xW has ones its selected w u region and zeros elsewhere. We then apply each mask M. to the frequency representation F(x,) by clement-wise multiplication, and reconstruct the corresponding spatial-domall image using the inverse Fourier transform:
faj F-1 (M, F(x)))
3

---

Published as a conference paper at ICLR 2026
Backhone
Stasd
REOROOR past ancoiiig silooce
Eacoket
Solect
Eacoder
Koepann-lageee Dynanici Sp
Pach Staked
Morid
Mack Riaanl eaage Vol.). M
Place Uizewvring Frespancy Shamars
Phanc EI Proeree Intisstaaiio
(b)
Figure 2: (a) PAE pipeline: MPA first initialize per-layer prompts. KLD then propagates prompts across layers via a shared Koopman operator, with a Lyapunov-style regularizer constraining error growth. (b) MPA pipeline: Frequency-domain transformations generate candidate masks, from which top ones are selected to build the initial prompt and propagate it across layers.
where @ denotes element-wiss multiplication. To evaluate the predictive utility of cach frequency region, we use the reconstructed images { to compute the task loss for each mask M,. We then rank all masks (M, ) in ascending order of their corresponding Lask, A lower loss implies that the masked frequency region retains more class-discriminative information for the task.
Phase -- Prompt Initialization. We select the top-I' masks, i.e., the T frequency masks with the lowest task losses Cusk,s, to initialize the first-layer visual prompt Pinir E 7xxd Each selected mask M, serves frequency shortcut, capturing localized spectral region that preserves strong -iseiiiicrimeeaivve information. To encode these frequency shortcuts into prompt tokens, we pro- ceed follows. For each selected mask M, (M,..- Mr} and each image xy (x1,-..,,7) in a mini-batch, we apply M, to the image's Fourier spectrum and reconstruct the filtered image Gt,j using Equation 2 Each ø.j then passed through the frozen patch embedding module to yield patch tokens [Pr.j.n BIxd]_ J==1, where N the number of image patches. To aggregate the discriminative content emphasized by each frequency shortcut, we employ token energy- meigh-weighhed pooling strategy. The intuition is that tokens with higher activation norms tend carry stronger semantic signals. Specifically, we compute each token's 's energy and normalize it across the batch:
en = |
Wt.j.n
(3)
Ey'=1
EN C.j at
We compute the representative token Aú and stack them to obtain the initial prompt matrix Pitit;
B N Pt
Wi.j. Pr.jan
pinit
E Txd
(4)
j=1 n=
To initialize deeper layers, we propagate Pinit through the frozen transformer encoder blocks (F,) 4-1 Starting from the first-layer, each block processes only the prompt and outputs the prompt for the next layer:
pinit E, (Pinit) for L.ii,,....44
(5)
This propagation transfers rreueency-aligned semantics through the transformer hierarchy, forming coherent, task-aware prompt initialization trajectory across all layers.
2.3 OPTIMIZATION VIA KOOPMAN- LYAPUNO- DISCRETE DYNAMICAL SYSTEM (KLD)
Following the MPA initialization, we use KLD establish cross-layer association of prompts (P, k. KLD is divided into two parts. Firstly, the cross- cross-layer association is established through the Koopman operator, which enables linear evolution in a lifted latent space, facilitating prediction and control from nonlinear data (Mezc)) 2013). Then, a Lyapunov-style regularizer is employed to limit the emor of the Koopman operator calculation across layers.
4

---

Published as a conference paper at ICLR 2026
Prompt Linear Projection. To establish cross- cross-layer associations, KLD first linearly projects the T prompt tokens at each layer into a shared latent space. We introduce a global learnable projection matrix U R where K is the dimensionality of the Koopman space. This matrix is initialized using Kaiming-uniforn initialization (He et al. 2015), and projects the prompt matrix P; E RTxd from layer into linear representation 2; IR X follows:
= P.U.
(6)
Within this shared latent space, we model prompt evolution using a global shared Koopman operator K IR kxx K which is initialized the identity matrix.
Koopman Operator Evolution.
Unlike conventional VPT variants that tune prompts independently per layer, we introduce cross-layer coupling via K. Specifically, we evolve the current state 2, into the next state 2 using:
Pirs
Encoder Layer;
Projoctiod
Rei
Original space (Independent)
Shared space (Association)
2,41 Z,
(7)
Illustration of Koopman evolution. In standard VPT, prompts at different layers are independent. In con- trast, KLD maps each layer's prompt into shared latent space, where a global shared Koopman operator K governs their evolution, enabling smooth cross-layer transitions.
Figure 3:
This formulation replaces per-layer independence with global consis- tent evolution process, as visualized in Figure
Learning the Koopman Operator. Effective evolution requires learning both U and K. The matrix U learns to map prompts into a space where transitions between layers are approximately linear, while K captures the shared layer-to-layer dynamics. To jointly learn these parameters, we introduce the Koopman consistency loss Lkp, which minimizes the difference between the predicted evolution 2,41 and the actual projected state 2;+1:
Cup
22+1
2 11
Σ -P,KK
where || |||F denotes the Frobenius norm. This loss provides explicit gradient flow across layers by coupling all prompts P = (P.) via the shared parameters U and K. The gradient with respect each prompt P, is:
0Lip AP:
- PUK)))
proceding-lnyer consistency: ()-1)=i
succeding-layer consistency; i-+(i+1)
This formulation reveals how each layer is pulled toward consistency with both its preceding and succeeding layers, enforcing global smooth and coherent prompt evolution
A Lyapunov-style Regularizer for Evolution Stability. fect and errors can accumulate across layers (Philipp et al. 2024), we introduce a Lyapunov-style regularizer Luab mitigate error growth and improve stability. Specifically, Lubb employs Lya- punoy function (zz) = tr(zQz' ). where Q E K K learable symmetric positive definite matrix. We regard the evolution as stable when the cross-layer error non-increasing, i.e., when associations between successive layers do not deteriorate (Haber & Ruthotto (2017). In other words, stability requires V(z+1)) < VZz)). This condition can be implemented as follows:
Since the linear approximation is imper-
L-1
Satab Σ
ma ((z+1)) - V(z:)).
(10)
As stability regularizer, Lruh penalizes increases in the evolution error between successive layers. The penalty is adaptive, and is applied only when cross-layer association degrades, so the evolution remains stable without excessive constraints
"See Appendix A.2 for full derivation.
"The gradient derivation in Appendix A.3
5

---

Published as a conference paper at ICLR 2026
Table 1: The speedup( x) and image classification accuracy (%) for ViT-Base/16 pretrained on su- pervised ImageNet-21k. Values are raw accuracy with PAE gains in parentheses.
VTAB-II lk
Method with PAE
Speedup(x)
FGVC
Natural
Specialized
Structured
Mean Total
Full Fine-tune
88.54
75.88
83.36
47.64
68.96
VPT (Jia et al. 2022
1.78
89.11 +1.91
78.48
+3.25
82,43 +2.09
54.98 +3.30
71.96 +2.88
E2VPT (Han et al.. 2023
1.65
89.22
+1.74
80.01
+1.38
84.43
+1.33
57.39
+2.34
73.94 +1.68
LPT (Dong et al 2023
1.44
89.94 +1.38
79.24
+1.77
83.40
+1.62
57.39
42.02
73.34
+1.81
VQT [Tu el al 2023
1.52
89.41 +1.21
79.46
+2.93
82.23
+3.11
57.47
+2.36
73.05
+2.80
VEPT Zeng ef al. 2024
1.27
89.24 +2.24
81.35
+0.72
84.93
+1.03
60.19
+0.77
75.39
+0.94
SA2VP Per el al. 2024
1.60
90.08 +1.12
80.97
+1.89
85.73
+0.85
60.80
+2.25
75.83
+1.66
ProVP Xu ct al 2025
1.19
89.56
+1.62
80.35
+1.98
84.07
+1.34
60.31 +1.41
74.91
+1.57
BPT Wang el al.
2025b
1.37
90.86 +1.35
80.24
+2.22
84.45
+1.88
60.39 +1.66
75.02 +1.92
Total Training Objective. objective function that combines the primary downstream task loss, Lrask, with our two dynamics- based regularizers:
Finally, our complete model is trained end-to-end by minimizing total
Liotll = Linsk + <<Cpp + B.
(11)
where a and (3 are hyperparameters that balance the enforcement of coherent dynamics and stability against the task-aware objective.
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Datasets We conduct experiments on 2 classification benchmarks: Fine-Grained Visual Classi- fication (FGVC) and VTAB-IK (Zhai et al. (009)). VTAB-Ik comprises 19 tasks across Natural, Specialized, and Structured domains, with constrained budget of 1,000 labeled training samples per task. The FGVC benchmark refers to collection of five fine-grained classification datasets. Additionally, we evaluate semantic segmentation performance on the ADE20K (Zhou et al. 2017 All 25 datasets follow the training and testing split protocols defined in VPT (Jia et al. 2022
Baseline Models To evaluate the effective- ness of PAE, we incorporate PAE into several state-of-the--at VPT variants, including struc-
Table 2: Results of ADE20K datasets with VIT-L. SS/MS denote single/multi-scall inference. Val- ues are mloU with PAE gains.
tured prompt designs (vPT (Jia al., (202)) E2VPT (Han et al.] (223). VQT (Tu et al. 2023), ProVP (Xu et al. 2025), BPT (Wang
Methods
Speedup(X) x)
mIoU-SS
mioU-MS
Full-tuning
47.60
49.18
et al.. 2025b)), adaptive prompting (LPT (Dong ctt al. 2023)), and perception-driven priors (VFPT Zeng et al.) 2024), SA2VP (Pei et al. 2.24 For each method, we report both its original performance and the enhanced version with PAE, denoted "Baseline +A", isolate the contribution of our framework. Then, we quantify convergence rate with speedup: the baseline epochs reach its best val acc divided by the epochs required after using PAE. Values > mean faster convergence. In addition, we include Full Fine-tuning, where all parameters are updated. The experiments are conducted by Transformer architectures: 3 variants (ViT-B/16, -L/16, and -H/14.) of ViT (Dosovitskiy et al.) 2020) and Swin-B (Liu et al. 202T), where are pre-trained on ImageNet- 21k (Deng al., (009). Moreover, we evaluate a sulf-supe ViT B/16 backbone: MAE (He et al. 2022). and segmentation ViT L/16 backbone: SETR (Zheng et al. 2021).
SPT-LoRA
45.40
47.50
VPT
1.29x
44.08 2.733
46.01 + 1.96
E2VPT
1.18x
44.61
2.32
46.56
2.84
VEPT
1.15x
45.32
2.75
47.17 2.09
Implementation Details In MPA, we set the window size w=16 with stride r=8. We randomly se- lected input mini-batch to discover frequency shortcuts. Then, the dimension of Koopman operator was set to 256. The weights were set to a = 0.5 for Lxp and B = 0.2 for Lusb [] Cross-entropy (CE) loss was used for Lusk- After adding PAE, the initial learning rate set 0.25, which was differ- ent from the larger learning rate of VPT variants. The batch size was set to 128. All experiments were conducted on an NVIDIA A800 GPU.
DDtailled descriptions and statistics of these datasets are provided in Appendix A.4 Moor ablation studies on window size, stride, loss weights are provided in Appendix A.5

---

Published as a conference paper at ICLR 2026
Col Loss Landscope
(b) Sharpoess (Rw))
(c) Condition Number
Surtace
abels (dtrerton)))
abdt (baseca
1000
VPT
2000
2r06
beta (dtrectivn-2)
(direction--)) 211
(direction-2)
VPT PAB
abhu (iteetarll
46
02yha direetac-11
Coaa
2010
2.5
beta (dinection2)
beta (dinection))
bata (dinection-)
Figure 4: Loss landscape comparisons (Li al. 2018) show (left right): 2-D loss contours, sharpness (max Hessian eigenvalue), anisotropy (Hessian condition number), and a 3-D surface colored by curvature.
3.2 RESULTS
The results the FGVC and VTAB-Ik benchmarks summarized in Table With only slight increase in preprocessing time, MPA completes initialization in just 74.17 seconds, roughly equivalent to 5.3 training epochs. Incorporating PAE consistently boosts performance across all settings, underscoring its versatility as a prompttagnostt enhancement with no inference-time over- head. For instance, integrating PAE into VPT (Jia et al.) 2022) raises FGVC accuracy on 91.02% (89.11%+1.. 11%++.11199999 %) and VTAB- 1k mean accuracy on 74.84% (71.96% +2.88% with convergence speedup of 1.78x. x. Similarly, when applied SA2VP (Pei et al.] 2024), improves VTAB-Ik performance on 77. 17.49% (75.83 (75.83%+1.66% ))), and achieves a 1.60x speedup in terms of convergence. Furthermore, adding PAE brings improvements on several VPT variants on ADE20K (Table [ Specifically, adding PAE to VPT, F2VPT, and VEPT increases mloU by about 2-3% under both single- and multi-scale evaluation, while also speeding up by roughly 1.15 to 1.29 x. These results show that PAE is still beneficial for dense prediction tasks such as semantic segmentation.
Furthermore, we visualize the loss landscape on 2D subspace and report curvature statistics in Figure Applying PAE results in substan- tially larger low-loss region with near-cireular contours, indicating reduced anisotropy. The Hessian-based sharpness map shows dimin- ished high-curvature rings when PAE is ap- plied, and the condition number heatmap dis- plays uniformly lower values, both suggesting more isotropic loss surface. The 3D surface plot further illustrates that VPT with PAE con- verges to wider and flatter minimum, whereas the bascline VPT remains sharp and narrow. Then, we employ
VPT
VPT PAE
Eapat
Epuats
Epuch 30
Epoch 50
Figure 5: Grad-CAM visualizations at early train- ing stages (epochs 5, 30, and 50) show that VPT+PAE rapidly concentrates attention on class- discriminative regions and stabilizes the saliency patterns, showing faster convergence than VPT.
(Selvaraju et al. 2017) to visualize the impact of integrating PAE into VPT on the CUB dataset, which consists of 200 fine-grained bird species. As shown Fig- ure 5 at epoch 5, VPT displays diffuse and non-discriminative attention, while VPT+PAE already focuses on task-relevant regions such as the wing and beak when classifying the Tennessee warbler. By epoch 30, VPT begins to localize the bird, but VPT+PAE delineates its full body with greater precision. At epoch 50. VPT+PAE produces sharper and more consistent saliency maps, indicating improved discriminative localization. These observations collectively indicate that using PAE leads to flatter, more stable minima, enhances discriminative localization, and accelerates convergence.
To assess the efficiency and scalability of PAE across diverse architectures and model sizes, we evaluate VPT, E2VPT, VFPT, and VPT+PAE across three VTAB- 1k groups on 4 backbones that vary in different seale and architecture (ViT-B/16, Swin-B, ViT-L/16, ViT-H/14) in Figure Across all three groups, VPT+PAE consistently improves over VPT every backbone and remains competitive
'Complete per-dataset are provided in Appendix A.7

---

Published as a conference paper at ICLR 2026
VPT
E2VPT
VEPT
4AT
VTAB-Natural
VTAB-Specialized
VTAB-Structuree
53
Test
78
76
(05.0) (85-791
(20830
(632M)
(65.000
(26.790
(304M)
(53270)
(05.30] (85-7M4
(30AN)
(632
Figure 6: VPT, E2VPT, VFPT, and VPT+PAE acToss three VTAB groups and four backbones (ViT-
B/16 + ViT-H/14). The shaded area marks the gap between VPT and VPT+PAE.
with better than E2VPT and VFPT, indicating that PAE scales smoothly from base to huge ViTs and transfers to other architectures, such as Swin-B.
Beyond performance, further analyze cross- layer prompt interactions by visualizing prompt CKA (Kornblith et al., 2019) across layers an MAE backbone, following Yoo et al. (2023). As shown in Figure 7 VPT and VFPT exhibit large blocks of high CKA, reflecting highly redundant prompts and weak depth dependent structure. GatePT (Yoo et al., 2023) slightly smooths these correlations but still maintains globally high similarity across most layers. In contrast, VPT+PAE produces a sharp diagonal band in the CKA matrix, where similarity is strongest locally and gradually decreases with layer distance. This pattern indicates pro- gressive, depth-aware evolution the prompt state rather than globally entangled represen- tation. The diagonal structure aligns with our formulation: single shared Koopman operator governs prompt dynamics and induces stable, forward-evolving trajectory across layers.
Porrpt screel ligers
Foorgtt scroec layees
5 10
11
VPT
VEPT
Poumpt CVA scress lipers
Fromps OVA scroes layerr
62
7 13
15
GasPPT
VPT
Figure 7: Prompt CKA
(Kornblith et al. 2019
across layers for different VPT variants MAE.
We additionally analyze how PAE behaves on classes with different levels of intra-class vari- ance (Cong et al., 2024) on CUB-200-2011. For intra-class variance and sort classes along the x-axis from low to high variance. Fig. M(a), we plot the per-class accu- racy of +PT+PAE against intra-class variance. We observe a mild negative Pearson correlation (corr -0.290), this shows high-variance classes tend to have lower accuracy, confirming that intra- class variance is good proxy for class difficulty. In Fig. (b), we plot the relative accuracy gain of VPT+PAE over vanilla VPT for each class, using the same ordering by intra-class variance. Here we observe a mild positive correlation (corr = 0.207), this shows although high-variance classes absolutely harder, they receive larger relative improvements from PAE. This indicates that PAE provides the greatest benefit on the difficult, high-variance categories where VPT struggles most.
CUB: cone 0.200
CUB comeo 207
Senueer on and san XPF
Ungy
each class, we compute
UP
(b)
Figure 8: (a) VPT+PAE per-class accuracy vs. intra-class variance. (b) PAE improvement over VPT vs. intra-class variance.
3.3 ABLATION STUDIES
We conduct ablation study to dissect the PAE framework and quantify the contributions of core components. Results are presented in Table 3 The study reveals two key findings. First, the MPA strategy is the most critical contributor to performance. When used in isolation, MPA yields a +2.06% improvement on the VTAB- mean score over the baseline. This gain significantly exceeds that of using only the Koopman consistency loss (Lap). Second, Zxp and the Lyapunov stability loss Luab
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation of PAE core elements on ViT-Base/16 using VPT (Jia et al.
2022) as baseline.
VTAB-IK lk
MPA(init)
KLD (Cip)
KLD (Cash)
FGVC
Natural
Specialized
Siruciared
Mean
89.11
78.45
82.43
54.98
71.96
89.63
81.35
84.07
56.64
74.02
90.56
80.27
83.28
55.82
73.13
90.78
81.46
83.85
57.96
74.42
91.02
81.73
84.52
58,28
74.84
Table 4; Ablation studies on ViT-B/16, using VPT with PAE: (Left) Comparison across different prompt initialization strategies. (Right) Variants of frequency shortcut injection.
(a) Prompt Iniitalization Strategies
(bi Pregneney-Shrtewn Injection Variants
Mothod
RGVC
Natural
Specialioed
VIAB-TK
Structured
Mean
Viriant
RGVC
Natral
Specialiced
VIAB-IC
Stractored
Mean
18.48
$4.98
Xavier
89.55
80.67
82.30
53.91
72.30
Copy
89,83
80.05
83.11
56.34
73.17
UnPro
90.23
80.46
83.18
57.89
73.85
Layer-wise
89.86
81.27
85.08
57.93
74.29
NPA (Ourr)
91.02
81.73
$4.52
58.28
74.84
Shared (Ours)
91,02
81.73
84.52
$8.28
74.84
are also essential and demonstrate strong synergy. While Exp alone yields notable improvement, adding Lusb results in further +1.29% gain. The full PAE framework, integrating all three compo- nents, achieves the highest performance, confirming that each part plays a complementary role in addressing the underlying optimization challenges.
To further validate the effectiveness of MPA, we compare it with several commonly used initial- ization strategies, as shown in Table 4a). For fair comparison, all methods use the Lxp and Lstab- Random initialization yields a VTAB- 1k mean score of 71.96% while Xavier ini- tialization (Xu et al.] 2025) slightly improves this to 72.30% but lacks task-aware aligmment. A stronger baseline uses features derived from
Table 5: Ablation on different input mini-batcher batches using VPT+PAE.
Mini bach
PGve
Natonol
Spectiliind
VIAB-IK
Structuen/
Moon
P p-value
1
91,02
51.73
$4.52
55.28
74.34
90.62
81.47
84.54
57.95
74.65
0.081
91.07
81.39
$4.25
57.92
74.51
0,092
1919)
84.96
66
0.009
2919)
90.64
81.34
84.14
57.91
74.46
0.008
Unsupervised Prototypes (Wang et al., 2024), achieving 73.85% by leveraging general-purpose visual representations. In contrast, MPA consistently outperforms all these alternatives, reaching 74.84%. This highlights its unique advantage: rather than relying on generic representations, MPA conducts task-awand search identify frequency shortcuts that are most relevant to the target task.
We further investigate how prompts are propagated across layers in MPA in Table 4b).. After generating frequency shortcut masks, MPA constructs the first-layer prompt and propagates it through the frozen backhone to initialize prompts in subsequent layers- achieving the best performance at 74.84% VTAB-Ik mean. This outperforms two alternative designs. First, copying the first-layer prompt to all layers yields 73.17%, dicating that building a layer-specific prompt hierar- chy is crucial. Second, performing layer-wise inde- pendent search achieves 74.29%, which still underper- forms MPA. These findings show the effectiveness of single-search and propagate strategy.
Ablation the Koupnun Operator Dimension
80
Tops Anomery
25
FDVC
Narza
Koopman Openator
Figure 9: Ablation on the Koopman oper- ator dimension in VPT+PAE on ViT-B/16.
With batch size and hyperparameters fixed (w=16,
8-8, B 128), examine the effect of randomly selected input batches on MPA, shown in Table 5 Batch B(1) is used as the baseline, and paired l-tests are performed against it. Deviations across task categories remain small: FGVC < 0.40, Natural S 0.39, Specialized V 0.44, and Structured 0.38, yiclding E [0.008, 0.092]. These results demonstrate that the randomness introduced by selecting different input batches for MPA has a negligible impact on performance.
Finally, we study the effect of the Koopman operator's latent dimension K. As shown in Figure K controls the expressivity of the linear dynamics model. The results show a clear trade-off: a small dimension (e.g., K 64) leads to underfitting and poor performance (67. 11%). As K increases, performance improves, peaking at K = 256 with a mean VTAB-Ik score of 74.84%. However, further increasing K to 384 degrades performance (68.24%, 24%). suggesting that overly large latent

---

Published as a conference paper at ICLR 2026
Walheer procett Mentlane
Lay WOPLD
Low SptreD
Figure 10: (a) Global Koopman spectra computed for the Natural, Specialized, and Structured sub- sets of VTAB-IK, Ik. (b) Comparison between the spectrum of the global Koopman operator (top) and that of the layer-wise operator at layer 7 (bottom). (c) The layer-wise design exhibits the individual spectra for each layer-specific operator.
space introduces optimization challenges and increases the risk of overfitting. Based on this analysis, we set K 256 as the default configuration.
ANALYSIS
According to classical linear systems theory. Eq is globally asymptotically stable the spectral radius p(K) (Guglielmi & Overton, 2011. In Koopman theory, eigenvalue magnitude dictates persistence: values near one decay slowly. while small ones vanish quickly (Budisié et al. 2012 Williams et al.) 2015). In practice, we analyze the spectrum via Kr Av. Specifically, Eqg 8 (LAp) is used in the shared latent space, and Eq TO (Leat) penalizes growth of the energy. These losses [q
favor contraction without rotation and concentrate the spectrum the positive real axis.
We first compare the global Koopman spectra learned by PAE on VTAB-IK benchmark from the Nat- ural, Specialized, and Structured groups As shown in Fig. 10 a). all three learned global operators are stable with p(K) < 1, but their spectra exhibit clear group-specific patterns. Specifically, Natu- ral tasks show moderate mean radius (mean|>) = 0.58), indicating that PAE only needs to gently reshape pretrained features when transferring to other natural-
tasks. Specialized tasks attain the largest mean radius (mean A\| = 0.67) with slow modes near unity, suggesting long effective memory for strong domain shifts. This helps cope with stronger domain shifts such as histopathol- ogy and remote sensing. In contrast, structured tasks yield the smallest radius (mean|>) |>| = 0.44), consistent with aggressive damping of texture variations. These group-dependent spectra reveal dynamical patterns in how PAE adapts its cue dynamics to different downstream tasks.
Next, we compare the spectrum of the global operator learned by PAE against layer-wise design, in which an independent learnable operator K, is assigned to each layer transition - i+ Fig. b) shows that the global operator concentrates eigenvalues along the positive real axis with p(K) < 1, ensuring coherent evolution. In contrast, the eigenvalue radius at layer 7 exceeds 3, reaching 3.68. As shown in Fig. c)) the layer-wise variant yields highly heterogeneous dynamics. Several layers exhibit large spectral radii (eigenvalue radius > 1), introducing mismatched time scales and unstable modes that are significantly harder to regularize.
5 CONCLUSION
In this paper. we reframe visual prompt tuning as a discrete dynamical system and introduce PAE, coupling MPA and KLD. MPA employs a supervision mechanism to find task-aware frequency short- cuts, which convert into initial prompts. KLD applies a shared Koopman evolution with a Lyapunov- style regularizer. This establishes cross-layer dependencies explicit, stabilizes gradients, accelerates convergence, and improves performance. Across multiple variants and image-llasiffiationn bench- marks, PAE improves accuracy and convergence. In particular, PAE remains wrmmpt-ggosttie drops into various VPT variants with no backbone changes and no inference-time overhead, offer- ing a practical and scalable path to improve visual prompt tuning. Limitation: MPA may be less suitable for low- or zero-label regimes unless appropriate surrogate signals are available.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marko Budišić, Ryan Mohr, and Igor Mezić. Applied koopmanism. Chaos: An Interdisciplinary Journal of Nonlinear Science, 22(4), 2012.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for 2020.
efficient on-device learning. NeurIPS,
Cong Cong. Shiyu Xuan, Sidong Liu, Shiliang Zhang, Maurice Pagnucco, and Yang Song, Decou- pled optimisation for long-tailed visual recognition. Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 1380-1388, 2024.
Cong Cong, Yang Song, Antonio Di leva, Angela Chou, Anthony J Gill, Enrico Coiera, and Sidong Liu. Flexible and scalable federated learning with deep feature prompts for digital pathology. 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Ieee, 2009.
Bowen Dong, Pan Zhou, Wangmeng Zuo, et al. Lpt: Long-tailed prompt tuning for image classifi- cation. In The Eleventh International Conference on Learning Representations, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learn- ing Representations, 2020.
Lei Fan, Dongdong Fan, Zhiguang Hu, Yiwen Ding, Donglin Di, Kai Yi, Maurice Pagnucco, and Yang Song. Manta: A large-scale multi-view and visual-text anomaly detection dataset for tiny objects. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25518- 25527, 2025a.
Lei Fan, Junjie Huang, Donglin Di, Anyang Su, Tianyou Song, Maurice Pagnucco, and Yang Song. Salvaging the overlooked: Leveraging class-aware contrastive learning for multi-class anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21419-21428, 2025b.
Zhengyi Fang, Yue Wang, Ran Yi, and Lizhuang Ma. Dropout mixture low-rank adaptation for visual parameters-efficien fine-tuning. In European Conference on Computer Vision, pp. 369- 386. Springer, 2024.
Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine-grained car detection for visual census estimation. Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
Nicola Guglielmi and Michael L Overton. Fast algorithms for the approximation of the pseudospec- tral abscissa and pseudospectral radius of matrix. SIAM Journal on Matrix Analysis and Appli- cations, 32(4): 166-1 192, 2011.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34(1):014004, 2017.
Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint 1.0iv:2307.13770,, 2023.
Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-awarr visual paramcter-efffiieet fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 11825-11833 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.

---

Published as a conference paper at ICLR 2026
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo. Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICMI, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference computer vision, pp. 709-727. Springer, 2022.
Zhaolu Kang, Junhao Gong, Jiaxu Yan, Wanke Xia, Yian Wang, Ziwen Wang, Huaxuan Ding, Zhuo Cheng, Wenhao Cao, Zhiyuan Feng, et al. Hssbench: Benchmarking humanities and social sciences ability for multimodal large language models. arXiv preprint arXiv:2506. 03922, 2025.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. Proc. CVPR workshop on fine- eite-grained visual categorization (FGVC), 2011.
Simon Kornllthh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pp. 3519- 3529. PMIR, 2019.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances neural information processing systems, 31, 2018.
Yiyang Liu, James Liang, Heng Fan, Wenhao Yang, Yiming Cui, Xiaotian Han, Lifu Huang, Dongfang Liu, Qifan Wang, and Cheng Han. All you need is one: Capsule prompt tuning with single vector. arXiv preprint arXiv:25 16670, 2025.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Zheda Mai, Ping Zhang, Cheng Hao Tu, Hong- You Chen, Quang-Huy Nguyen, Li Zhang, and Wei- Lun Chao. Lessons and insights from unifying study of parameter-efficien fine-tuning (peft) in visual recognition. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14845-14857, 2025.
Gabriele Merlin, Vedant Nanda, Ruchit Rawal, and Mariya Toneva. What happens during finetuning of vision transformers: An invariance based investigation. In Conference on Lifelong Learning Agents, pp. 601-619 PMLR, 2023.
Igor Mezić. Analysis of fluid flows via spectral properties of the koopman operator. Annual review fluid mechanics, 45(1):357-378, 2013.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV. pp. 69-84. Springer, 2016.
Wenjie Pei, Tongqi Xia, Fanglin Chen, Jinsong Li, Jiandong Tian, and Guangming Lu. Sa²vp: Spatially segned-and-adappee visual prompt. In Proceedings of the AAAI conference on artificial intelligence, pp. 4450-4458, 2024.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.002: 00247, 2020a.
12

---

Published as a conference paper at ICLR 2026
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations, pp. 46-54, Online, 2020b. Association for Computa- tional Linguistics.
Friedrich Philipp, Manuel Schaller, Karl Worthmann, Sebastian Peitz, and Feliks Nüske. Error bounds for kernel-based approximations of the koopman operator. Applied and Computational Harmonic Analysis, 71, 2024.
Li Ren, Chen Chen, Liqiang Wang, and Kien Hua. Da-vpt: Semantic-guiin guiiied visual prompt tuning for vision transformers. In Proceedings of the Computer Vision and Pattern Recognition Confer- ence, pp. 4353 4363, 2025.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- gradient-based based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626 2017.
Daniel Shevitz and Brad Paden. Lyapunov stability theory of nonsmooth systems. IEEE Transac- tions automatic control, 1910)1110-1111, 1994.
Kihyuk Sohn, Huiwen Chang, José Lezama, Luisa Polania, Han Zhang, Yuan Hao, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 19840-19851, 2023.
Cheng-Hao Tu, Zheda Mai, and Wii-Lun Chao. Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7725-7735, 2023.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print fine-grained dataset collection. In Proceedings of the Computer Vision and Puttern Recognition Conference, 2015.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Shunxin Wang, Raymond Veldhuis, Christoph Brune, and Nicola Strisciuglio. What do neural networks learn in image classification? frequency shortcut perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1433-1442, 2023.
Shunxin Wang, Raymond Veldhuis, and Nicola Strisciuglio. Do imagenet-trained models learn shortcuts? the impact of frequency shortcuts on generalization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 55198-25207. 2025a.
Yuzhu Wang, Lechao Cheng, Chaowei Fang, Dingwen Zhang, Manni Duan, and Meng Wang. Revis- iting the power prompt for visual tuning. Proceedings of the 41st International Conference on Machine Learning, pp. 50233-5024 2024.
Yuzhu Wang, Manni Duan, and Shu Kong. Attention to the burstiness in visual prompt tuning! In International Conference on Computer Vision (ICCV), 2025b.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pultern recognition, pp. 139-149, 2022.
Matthew O Williams, Ioannis G Kevrekidis, and Clarence W Rowley. A data driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25(6):1307-1344 2015.
13

---

Published as a conference paper at ICLR 2026
Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang. Yuxiang Wei, Jihun Hamm, and Min Xu. Visual instance-awand aware prompt tuning. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 2880-2889 2025a.
Xi Xiao, Yunbei Zhang, Yanshuh Li, Xingjian Li, Tianyang Wang, Jihun Hamm, Xiao Wang, and Min Xu. Visual variational autoencoder prompt tuning. arXiv preprini arXiv:2503. /7650, 2025b.
Xi Xiao, Yunbei Zhang, Lin Zhao, Yiyang Liu, Xiaoying Liao, Zheda Mai, Xingjian Li, Xiao Wang, Hao Xu, Jihun Hamm, et al. Prompt-based adaptation in large-scale vision models: A survey. arXiv preprint 1rX 02222510.111111 13219, 2025c.
Chen Xu, Yuhan Zhu, Haocheng Shen, Boheng Chen, Yixuan Liao, Xiaoxin Chen, and Limin Wang- Progressive visual prompt learning with contrastive feature re-formation. International Journal of Computer Vision, 133(2):511-55) 526, 2025.
Wenhao Xu, Rongtao Xu, Changwei Wang. Shibiao Xu, Li Guo, Man Zhang, and Xiaopeng Zhang. Spectral prompt tuning: Unveiling unseen classes for zero-shot semantic segmentation. In Pro- ceedings of the AAAI conference on artificial intelligence, pp. 6369-63399 6377, 2024.
Seungryong Yoo, Eunji Kim, Dahuin Jung, Jungheom Lee, and Sungroh Yoon. Improving visual prompt tuning for self-supervised vision transformers. In International Conference Machine Learning, pp. 40075-40092. PMLR, 2023.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? NeurlPS, 27, 2014.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficien fine-tuning for transformer-bet based masked anguage-modeht In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Short Pupers), pp. 1-9, 2022.
Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huangg, Ying Nian Wu, and Dongfang Liu. Visual fourier prompt tuning. Advances in Neural Information Processing Systems, 37:5552-55888 2024.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910008677, 2019.
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: baseline for network adaptation via additive side networks. In ECCV, pp. 698-714. Springer, 2020.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, pp. 649- 666. Springer, 2016.
Sixiao Zheng, Jiachen Lu. Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-sequeee perspective with transformers. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 6881-6890, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017
Shihao Zhou, Jinshan Pan, Jinglei Shi, Duosheng Chen, Lishen Qu, and Jufeng Yang. Sceing the unseen: A frequency prompt guided transformer for image restoration. In European Conference on Computer Vision, pp. 246-264. Springer, 2024.
Zhijie Zhu, Lei Fan, Maurice Pagnucco, and Yang Song. Interpretable image classification via non- parametric part prototype learning. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pp. 9762-9771, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USES OF LLM
We used large language model (LLM) only for language polishing (grammar, clarity, and style). All ideas, analyses, and conclusions are the authors' own, and the authors conducted the final review and approval.
A.2 GRADIENT FOR THE KOOPMAN CONSISTENCY LOSS
Here we provide the detailed derivation for the gradient of the Koopman consistency loss Lup with respect an intermediate prompt P.. The loss given by:
Exp Σ
Zi+1
K F
(12)
P.+1
8,12
The prompt P. (and its Koopman representation 2; P.U) influences two adjacent terms in this
sum: the term for the transition from to i, which we denote 1 = ||z Zi 2: K| and the term for the transition from i to + 1, denoted 4, 2 K
The gradient is the sum of the gradients of these two terms. First, we compute the gradient with respect to 2,
04-1 2(z, -K-
(13)
04,
= 2(z++1 Z.KK
(14)
Oz.
Combining these gives the full gradient with respect Z₁:
dLep 2(z: 2 K) - 2(z,+1 - 2,KjK = 2(z, - -2;) - 22(22+1 2/41)
(15)
Oz.
Finally, using the chain rule with z; P,U, obtain the gradient with respect to the prompt P;:
DLxp ÖCip dz, [2(z: 2.) - 2(2;+1 - 2.+ K
UT.
(16)
0P: Oz, OP;
This final gradient provides the direct supervisory signal used to update the prompts.
A.3 GRADIENT FOR THE LYAPUNOV STABILITY REGULARIZER
The Lyapunov stability regularizer is defined as:
Stab
max(0, AV),, where AV, è V - V(z.).
(17)
1=1
The Lyapunov function is Viz) = tr(zQz T). Its gradient with respect to z is OV (z)/000 = 2zQ, given that Q symmetric.
To handle the non-differentiaabiiity of the max(0, æ) function at x = 0, we can use an indicator function the derivation. Let 17: 1 (av>>)) be an indicator that if the error increases from step i + 1, and 0 otherwise. The loss can be expressed Latab = Σl_ AVV
Let's compute the gradient with respect to an intermediate state z; (where < i < L). The state z; influences two terms in the sum: 7)- AV.1- Th- (V=((((z,)) -V and 1/ AV; (T(VV (V(4+)) (z,)). The gradient therefore:
dLttab
i
M/ (V(z.) V(zz
1/: (V(z++)) (V(Z+t))- V(2i))
Oz:
Oz
Oz.
11:-1 (V(z;) (V(z))
(18)
On. -21/-12,00 21112 Q
82,
2((T-11 1): 11:22 Q
15

---

Published as a conference paper at ICLR 2026
This resulting gradient for z; is non-zero only if the stability condition changes at the boundary of step i (i.e., violated for step but not for step i, or vice-versa).
Finally, using the chain rule with 2, P,U, obtain the gradient with respect to the prompt P;:
Stub Stuu oz. 2(k 7/) )1)
(19)
OP;
iz; oP;
This provides the corrective signal to update the prompts, active only when necessary to enforce the constraint.
A.4 DETAILED DATASET DESCRIPTIONS
We provide comprehensive statistics for the datasets used in our evaluation, corresponding to the experimental setup described in the main text.
Fine-Grained Visual Classification (FGVC), This benchmark includes five datasets that require
distinguishing between highly similar subordinate categories. The specific data splits are as follows:
CUB-200-2011 (Wah et al.
2011): 200 classes (5,394 train, 600 val, 5,794 test).
NABirds (an Horn al.
2015 55 classes (21,536 train, 2,393 val, 24,633 test).
Oxford Flowers-102 Nilsback Zisserman 2008 102 classes (1,020 train, 1,020 val, 6,149 test).
Stanford Dogs (Khosla et al.
2011): p: 120 classes (10,800 train, 1,200 val, 8,580 test).
Stanford Cars (Gebru al.
2017) 196 classes (7,329 train, 815 val, 8,041 test).
VTAB-1k Benchmark. The Visual Task Adaptation Benchmark (VTAB-1K) (Zhai et al. 2019) limits the label budget to 1,000 training samples per task (800 for training, 200 for validation). consists of 19 tasks grouped into three categories:
Natural: CIFAR- 100 (10,000 test), Caltech101 (6,084 test), DTD (1,880 test), Flowers102 (6, 149 test), Pets (3,669 test), SVHN (26,032 test), and SUN397 (21,750 test).
Specialized: Patch Camelyon (32,768 test), EuroSAT (5,400 test), RESISC45 (6,300 test), and Retinopathy (42,670 test).
CLEVR/count (15,000 test), CLEVR<distance (15,000 test), DMLab (22,735 test), KITTI/distance (711 test), dSprites/lccatin (73,728 test), dSprites/orientation (73,728 test), SmallNORBBazimitt (12,150 test), and SmallYORB/ellvatioon (12, 150 test).
Structured:
Semantic Segmentation. For dense prediction tasks, we utilize the ADE20K (Zhou et al 2017) dataset. Train/val splits were followed the same split protocol as in main experimental setup.
A.5 ABLATION STUDIES ON HYPERPARAMETERS
As shown in 6 the frequency- -rid hyperparameters show that (w=16, r=8) attains the highest cross-benchmark Mean 74.84 and is best (or tied) on FGVC 91.02 and Natural 81.73. Although per-track optima differ Speccalzzed peaks at (16, 12) with 85.07 and Structured peaks at (24, 12) with 59.11. we choose (w=16, 8= as the final configuration because maximizes the Mean, which is our selection criterion across benchmarks. Unless otherwise stated, subsequent experiments use a=0.5, 3=0.2 and (w=16, r=8).
As shown in Table 7 introducing moderate loss weights consistently improves over the no-auxiliary baseline (a=3=0, Mean = 71.96). 3. The best average appears at a=0.5, 3=0.2 with Mean = 74.84 (+2.88), alongside gains all tracks: FGVC 91.02 (+1.91), Natural 81.73 (+3.25), Specialized 84.52 (+2.09), and Structured 58.28 (+3.3)). Further increasing a (e.g., a=0.9, 3=0.3) degrades the average (73.52), indicating over-regularization.
16

---

Published as a conference paper at ICLR 2026
Table 6: Ablation on frequency window size w and stride r for ViT-B/16 with VPT+PAE. Bold
denotes the best in column.
VIAB-IK
w
FGVC
Natural
Specialized
Structured
Meun
89.28
79.76
82.71
55.29
72.59
16
91.02
81.73
84.52
58.28
74.84
16
12
90.37
79.84
57.73
74.21
12
90.41
80.43
82.96
59.11
74.16
16
90.15
79.97
83.56
58.62
74.05
Table 7: Ablation on loss weights (a, 3) for ViT-B/16 with VPT+PAE. Bold denotes the best in column.
VTAB-IK B-I-
a
FGVC
Natural
Specialzed
Structured
Mcan
0.0
0.0
89.11
78.48
82.43
54.98
71.96
0.1
0.1
89.58
79.02
83.01
55.67
72.57
0.3
0.1
90.72
80.31
84,24
57.83
74.13
0.5
0.2
91.02
81.73
84.52
58.28
74.84
0.7
0.2
90.63
82.18
84.17
58.09
74.81
0.9
0.3
89.37
79.95
82.90
57.71
73.52
A.6 SENSITIVITY ANALYSIS OF CONVERGENCE SPEED.
To rigorously evaluate the robustness of our hyperparameter selection, we analyze the impact of the regularization weights (a, 33 on the convergence speed across four distinct benchmarks: FGVC, VTAB-Natural, TTABBSSpciallied, and VTAB-Structured. As illustrated in Figure II a consistent U-shaped correlation is observed between the regularization strength and the number of epochs required for convergence across all task groups
Specifically, the experimental results reveal three key regimes centering around an optimal configu- ration. The fastest convergence is uniformly achieved at the "Sweet Spot" setting of (a 0.5, B 0.2). In this region, the stability induced by PAE allows for aggressive optimization, reducing the training duration by approximately 40% 50% compared to the baseline (a 0, 0). Deviating from this optimum compromises efficiency. In the Under-regularization regime (e.g.. 0.1, 0.1), weak constraints fail to effectively suppress high-frequency gradient oscillations, leaving the optimization trajectory jagged and resulting in negligible epoch reduction. Conversely, Over- regularization (e.g., 0.9, 0.3) leads to sharp rebound in training costs due excessive constraints. This is particularly evident in the FGVC group, where "over-smoothingg" prompts detrimentally fects the learning of fine-grained discriminative features, prolonging convergence 96 epochs.
Notably, the Natural and Specialized groups exhibit the steepest improvements, benefiting most from the stabilized dynamics. While the Structured tasks show flatter improvement curve due to their inherent optimization difficulty, they still achieve optimal efficiency at the same parameter configuration, confirming the universality of our default setup.
A.7 FULL EXPERIMENTAL RESULTS
report per-dataset top-1 accuracy and convergence epoch for all evaluated methods, including baseline VPT variants and their PAF-augmented counterparts. For VTAB- 1k, we provide per-task scores for every task: Natural in Figure 9 Specialized Figure I0 and Structured in Figure IT] along with the group means and the overall mean. Training and evaluation settings match those in the main text unless otherwise noted.
For FGVC in Figure &
We compare both variants of VPT with other commonly used fine-tuning protocols:
(a) Fully fine-tume all backbone and classification head parameters.
(b) Methods that focus on the classification head. They treat the pre-trained backbone as a feature extractor, whose weights are fixed during tuning:
Linear: only linear layer the classification head.
17

---

Published as a conference paper at ICLR 2026
Optinal Region
Epochs Convergence
70
60
50
FGVC
40
IAB--aaural
VTA Speciaize
Seucttred
(0.00.00)
(0.1.0.1)
(0.3.0.1)
(0.5.0.2)
(0.7.02)
(0.9.0.3)
Hyperparameters (a, B)
Figure 11: Sensitivity analysis of convergence speed with respect to hyperparameters (a, B).
Partialft-A: fine-tune the last k layers of the backbone while freezing the others, adopted in (Yosinski et al. 20144 Zhang et al.) [016)] Noroozi & Favaro 2016 He et al. 2022).. This redefines the boundary between the backbone and the classification head.
Mlp-k: utilize multilayer perceptron (MLP) with le layers, instead of linear layer, the
classification head.
(c) Methods that update subset of backbone parameters add new trainable parameters to the backbone during fine-tuning:
Sidetune (Zhang et 2020): train "side" network and linearly interpolate between pre-
trained features and side-tuned features before feeding them into the head.
Bias (Cai et al. 2020 Zaken et al. 2022): fine-tune only the bias terms pre-trained backbone.
Adapter (Houlsby al. 2019 Pfeiffer et al.
2020a bb: insert new MLP modules with
residual connections inside Transformer layers.
LoRA (Hu et al. [2022] He et al. 2023 Fang et al. 2024): insert low-rank trainable matrices
into the projection layers while freezing the original backbone weights, enabling parameter- efficient adaptation.
18

---

Published as conference paper at ICLR 2026
Table 8: FGVC per-task results for ViT-Base/16 pretrained on supervised ImageNet-2Ik.
FGVC Oxford Flowers
Convergence
VIT- VTT-Base/16
Mcan
CUB-200-2011
NAbirds
Stanford Dogs
Stanford Cars
epoch
FULL
87.3
82.7
98.8
89.4
84.5
88.54
LINEAR
85.3
75.9
97.9
86.2
51.3
79.32
PARTIAL-1
82.63
MLP-2
85.7
77.2
98.2
85,4
54.9
80,28
MLP-3
85.1
77.3
97.9
84,9
53.8
79.80
MLP-5
84.2
76.7
78.71
MLP-9
83.2
76.0
77.31
SIDETUNE
84.7
75.8
96.9
85.8
48.6
78.35
BIAS
88.4
84.2
98.8
91.2
79.4
88.41
ADAPTER-250
87.2
84.3
ADAPTER-64
87.1
84.3
98.5
85.67
ADAPTER-8
87.3
84.3
98.4
88.8
68.4
85,46
LoRA
88.3
85.6
99.2
91.0
89.46
SPELLRA
88.6
DM-LoRA
89.8
86.6
99.5
91.8
85.7
90.68
VPT
88.5
84.2
99.0
90.2
83.6
89.11
92
+
89.7
87.4
99.2
92.11
86.8
91.02
54
E2VPT
89.1
84.6
82.8
E2VPT P&E
90.3
87.4
99.3
91.4
86.4
90,96
58
LPT
89.6
86.
99.0
90.9
84.2
89.96
87
LPT PAE
90.8
87.6
91.32
VQT
88.9
85.5
99.2
90.2
83.1
89.41
VQT + PAE
90.0
86.6
99.1
91.5
85.9
90.62
S6
VEPT
88.7
84.5
99.1
90.4
89.24
76
VEPT PAE
90.5
88.4
99.3
92.7
86.5
91.48
SA2VP
89.1
85.8
99.3
92.1
84.1
90.08
81
SA2VP PAE
89.3
88.
99.5
92.0
87.1
91,20
50
ProVP
ProVP PAE
90.4
87.3
99.2
92.3
86.7
91.18
70
HPT
90.2
87.5
99.7
90.1
86.8
90.86
93
BPT PAE
91.2
89.0
99.6
91.9
92.21
Table 9: VTAB- Natural per-task per-task results for ViT-Base/16 pretrained on supervised ImageNet-21k:
VTAB- Natural
Convergence
VYT-Base/16
Mcan
CIFAR-100
Caltech101
DTD
Flowers102
SVHN
Sun397
epoch
FULL
68.9
87.7
64.3
97.2
86.9
87.4
38.8
75.88
LINEAR
63,4
85.0
63.2
97.0
86.3
36.6
51.0
68.92 98.93
PARTIAL-I
66.8
85.9
62.5
97.3
85.5
37.6
50.6
69.44
63.2
97.6
MLP-3
63.8
84.7
62.3
97.4
84.7
32.5
49.2
67.80
MLP-5
59.3
84.4
59.9
96.1
84.4
30.9
46.8
65.98
MLP-9
53.1
80.5
53.9
95.1
82.6
24.4
43.7
61.90
SIDETUNE
60.7
60.8
53.6
95.5
66.7
34.9
35.3
58.21
BIAS
72.8
87.0
97.5
51.4
73.30
ADAPTER-255
74.1
86.1
63.2
97.7
87.0
34.6
50.8
70.50
ADAPTER-64
74,2
85.8
62.7
97.6
87.2
36.3
50.9
70.65
ADAPTER-8
74.2
85.7
62.7
97.8
87.2
36.4
50.7
70.67
LoRA
67.1
91.4
98.8
54.0
79.46
SPT-LORA
73.5
93.3
72.5
99.3
91.5
87.9
55.5
81.93
DM-LoRA
74,0
90.7
73.9
99.3
92.2
91.1
56.4
82.51
VPT
78.8
90.8
65.8
98.0
88.3
78.1
49.6
78.48
81
VPT + PAE
79.5
92.3
70.4
98.6
91.7
88.2
51.4
81.73
E2VPT
78.6
89.4
98.2
52.3
80.01
E2VPT +PA
79.1
91.3
69.4
98.0
89.8
87.6
54.5
81.39
LPT
77.2
89.9
66.4
97.8
89.1
83.4
50.9
79.24
88
LPT + PAE
78.8
91.5
69.7
98.0
91.3
86.1
51.7
81.01
61
VQT
77.9
88.4
68.2
98.1
82.8
51.2
79.46
VQT
80.3
91.0
73.6
99.2
85.7
54.4
82.39
VEPT
80.7
91.4
69.4
99.3
90.3
85.6
52.7
81.35
77
VEPT + PAE
82.1
91.8
69.5
99.6
91.7
87.2
52.6
82.07
62
SA2VP
73.0
91.9
70.5
99.1
90.8
84.7
56.8
80.97
SA2VP
77.5
99.6
56.5
ProVP
75.2
91.7
68.7
99.0
89.8
84,5
53.6
80.35
ProVP + PAE
80.6
92.8
71.3
99.4
89.3
86.7
56.2
82.33
74
BET
74.6
90.8
69.4
99.5
90.2
84.5
52.7
80.24
BPT PAB
82.46
19

---

Published as a conference paper at ICLR 2026
Table 10: VTAB-IK Specialized per-task results for ViT-Base/16 pretrained on supervised
ImageNet-21k.
VTAB IK Sperialized (4)
Convergence
ViT-Base/16
Patch Camelyon
EuroSAT
Resise45
Retinopathy
Mean
epoch
79.7
73.9
83.36
LINEAR
78.5
87.5
68.6
74.0
77.16
PARTIAL-I
78.6
89.8
72.5
73.3
78.53
MLP-2
74.3
73.2
75.86
MLP.3
77.0
88.0
70.2
56.1
72.83
MLP-5
73.7
87.2
64.8
71.5
74.31
MLP-9
78.5
72.3
73.49
SIDETUNE
58.3
61.0
68.12
BIAS
78.7
91.6
72.9
69.8
78.25
ADAPTER-22 256
76.3
88.0
73.1
70.5
76.98
ADAPTER
76.3
70,9
77.10
ADAPTER-8
76,9
89.2
73.5
71.6
77.80
LoRA
84.9
95.3
84.4
73.6
84.55
SPT-LORA
85.7
75.9
85.93
DM-LORA
85.6
96.5
87.0
76.1
86.30
VPT
81.8
96.1
83.4
68.4
82.43
69
VPT
83.1
72,5
84,52
34
E2VPT
82.5
96.8
84.8
73.6
84.43
90
E2VPT + PAE
83.6
97.4
86.1
75.9
85.76
56
71.8
LPT PAE
82.8
96.4
85.3
75.6
85.02
55
VQT
81.4
94.8
84.2
72.5
82.23
78
VQT
83.5
96.
74.9
VEPT
83.5
96.5
84.4
75.4
84,93
72
VEPT + PAE
83.4
97.8
85.9
76.7
85.96
58
SAZVP
86.0
75.2
SA2VP PAE
85.7
97.5
86.9
76.2
86,58
53
ProVP
84.6
95.7
82.4
73.6
84.07
87
ProVP
84.8
96.1
75.7
85.4]
73
BPT
85.8
96.2
81.5
74.9
84.45
88
BPT PAE
87.4
97.6
84.1
76.2
86.33
60
Table 11: VTAB-1k Structured per-task results for ViT-Base/16 pretrained on supervised ImageNet-
21k.
ViT-Base/16
Cleve! count
Clevn' distance
DMLab
KITTI/ distance
VTAB- IK Structured dSprites locatoo
dSpriseed orientation
SmalINORB/ azimuth
SmalINORB/ elevation
Mean
Convergence epach
FULL
56.3
58.6
41.7
65.5
57.5
46.7
25.7
29.1
47.64
LINEAR
55.4
PARTIAL-I
41.5
34.3
33.9
61.0
31.3
32.8
16.3
22.4
34.17
MLP-2
45.2
31.6
31.8
55.7
30.9
24.6
16.6
23.3
32.47
58.1
12.9
24.8
30.62
MLP.S
50.8
32.3
31,5
56.4
7.5
20.8
14.4
20.4
29.23
MLP-9
54.0
16.2
SIDETUNE
27.6
22.6
31.3
$1.7
8.2
144
21.8
23.41
BIAS
61.5
55.6
32.4
55.9
66.6
40.0
15.7
25.1
44.09
ADAPTER-64
42.9
39.9
30.4
$4.5
31,9
25.6
13,5
21.4
32.51
ADAPTER-8
56.4
30.4
13.2
22.0
33.09
LORA
82.9
49.8
78.5
75.7
47.1
31,0
44.4
59.83
T-LRA
84.4
67.6
52.5
$2.0
$1.0
$1.1
30.2
41.3
61.26
DM-LORA
VPT
68.5
60.0
46.3
72.8
73.6
47.9
32.9
37.88
$4.98
78
VPT PAB
73.4
61.7
16.9
75.8
77.9
53.6
34,3
42.6
58.28
56
EZYPT
71.7
47.9
75.8
80.8
48.1
31.7
41.9
57.39
E2VPT +PAE
72.9
63.7
49.4
78.5
82.5
$2.9
33.8
44.3
59.73
60
LPT PAB
74.8
49.6
75.8
80.6
54.7
32.9
43.4
59.41
60
VQT
71.8
62.6
46.4
73.7
79.3
50.8
32.5
42.6
$7.47
83
VQT
77.2
VEPT
75.8
63.2
48.3
79.3
81.5
36.0
34,1
43.4
61.19
74
VEPT + PAB
76.9
65.2
48.1
79.7
82.4
$5.2
36.5
43.7
60.96
56
79.9
SAZVP + PAB
78.2
72.5
53.3
79.6
86.4
$4.5
34.1
45.8
63:05
51
ProVp
75.5
67.2
49.3
77.6
83.7
51.9
33.8
43.5
60.31
84
ProVP- PAB
77.3
71.6
51.5
76.8
83.5
34.7
44.7
61.72
BPT
76.1
65.3
48.9
77.8
82.5
53.2
34.7
44.6
60.39
80
20


Reviews:
[{"review_id": "gdxzwRyVOW", "reviewer": "Reviewer_Z7w9", "summary": "This paper introduces Prompt-Agnostic Evolution (PAE), a framework designed to stabilize and accelerate training in Visual Prompt Tuning (VPT) for Vision Transformers. The authors identify that existing VPT variants suffer from unstable gradients, including shallow-layer stagnation and deep-layer oscillations. To address this, they propose two key components: 1) Modal Pre-Alignment (MPA): A frequency-domain initialization that aligns prompts with task-relevant frequency “shortcuts.” 2) Koopman-Lyapunov Discrete (KLD) system: A shared dynamical model where prompts evolve across layers under a Koopman operator with Lyapunov-based regularization for stability.\nExperiments on FGVC and VTAB-1k benchmarks show 1–3% accuracy improvements and 1.5× faster convergence in terms of the number of required epochs.", "questions": "Please see above", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel formulation: Reframing prompt tuning as a dynamical system using Koopman theory and Lyapunov stability is novel and mathematically grounded.\n\n2. Comprehensive analysis: The paper clearly diagnoses VPT training instability through layer-wise gradient visualizations and supports it with quantitative results.\n\n3. Strong empirical performance: PAE consistently improves various VPT baselines, showing strong generalization across tasks and benchmarks.\n\n4. Prompt-agnostic applicability: The method is modular, adding no inference-time overhead and requiring no backbone modification.\n\n5. Clear ablations: Ablation studies demonstrate the complementary roles of MPA and KLD, and verify robustness against random initialization and batch selection.", "weaknesses": "- Motivation clarity: While the dynamical-system framing is novel and interesting, the necessity of such complexity for solving gradient oscillation may be overstated. Simpler temporal regularization could have been compared. For example, could simpler smoothing (e.g., temporal moving average across layers) achieve comparable stability?\n\n- Dependence on frequency bias: MPA relies on identifying “frequency shortcuts,” which may not exist or be stable in non-natural image domains, limiting transferability.\n\n- Missing comparison on across-layer effects: The paper does not cite prior work such as [ref1], which examines how prompts interact across layers. A more direct comparison and analysis between that study’s findings and the proposed PAE framework would strengthen the discussion.\n\n[ref1] Improving Visual Prompt Tuning for Self-supervised Vision Transformers, ICML 2023\n\n- Questionable real-world efficiency: Although Fig. 1(a) and Table 1 show faster convergence in terms of epochs, the overall efficiency claim may be overstated. When accounting for MPA initialization overhead and the extra hyperparameters (α, β, K, w, r) that expand the tuning space, the total wall-clock time might actually increase. Hence, the practical benefit in large-scale or hyperparameter-sensitive scenarios remains uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YzFwRiRneh", "reviewer": "Reviewer_PEzU", "summary": "This paper addresses a core instability issue in Visual Prompt Tuning (VPT), a fine-tuning method for Vision Transformers (ViTs). The authors observe that many VPT variants suffer from unstable training processes, characterized by gradient oscillations and a cross-layer mismatch phenomenon, where prompts in shallower layers stagnate while those in deeper layers oscillate to compensate. This ultimately slows down convergence and hinders optimal performance. The paper identifies two root causes for this problem: task-agnostic prompt initialization and the independent, uncooperative optimization of prompts at each layer. To resolve this, the authors propose Prompt-Agnostic Evolution (PAE), a framework that treats prompt tuning as a dynamical system. PAE consists of two novel components. The first is Modal Pre-Alignment (MPA), which provides a task-aware initialization by identifying the most discriminative frequency shortcuts for a given task and using them to generate initial prompts. The second is the Koopman-Lyapunov Discrete Dynamical System (KLD), which governs the prompt optimization. It uses a shared Koopman operator to enforce a coherent linear evolution for prompts across multiple layers within a shared latent space, and a Lyapunov-style regularizer to ensure this evolution remains stable. In experiments on the FGVC and VTAB-1k benchmarks, applying PAE to various VPT methods consistently improved accuracy by 1-3% and accelerated convergence by an average of 1.48x. As a prompt-agnostic module, PAE can be integrated into existing methods without modifying the model's backbone and has zero overhead at inference time, making it a practical and effective solution.", "questions": "I would appreciate your response to the points raised in the \"Weakness\" section.\n\nAdditionally:\n- I would like to know how the convergence speed varies with changes in the hyperparameters alpha and beta.\n- I am interested in the performance differences based on intra-class variance. It would be helpful to see the difference in performance gains between the best- and worst-performing classes or class groups in the dataset.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The primary strength of this research is its novel conceptualization of prompt tuning as a dynamical system, providing a principled framework to address the observed training instabilities. By applying the Koopman operator and Lyapunov stability theory, it moves beyond empirical heuristics and introduces an explicit mechanism to coordinate prompt updates across layers, directly tackling the optimization mismatch problem. Another key innovation is the Modal Pre-Alignment (MPA) strategy. This method effectively solves the cold start problem in prompt tuning by using a task-aware initialization based on frequency-domain analysis. By identifying the frequency shortcuts already utilized by the pre-trained backbone, MPA provides initial prompts that are well-aligned with the task objective from the outset, which emerged as the single largest contributor to performance gains. Finally, the PAE framework demonstrates remarkable robustness and practicality. Its prompt-agnostic design allows it to be seamlessly integrated as a plug-and-play module into numerous state-of-the-art VPT variants, consistently improving performance in all cases. This proves that PAE addresses a fundamental weakness in the VPT paradigm. The comprehensive empirical validation, which includes not only accuracy metrics but also insightful loss landscape and Grad-CAM visualizations, provides strong evidence for its effectiveness. The fact that there is zero inference-time overhead further solidifies its value for real-world applications.", "weaknesses": "Assumptions of the KLD Framework: The Koopman-Lyapunov Discrete Dynamical System (KLD) assumes that the prompt dynamics can be effectively modeled by a single, global, linear operator. This could create a representational bottleneck for complex tasks where different dynamics in shallow and deep layers might be more beneficial. The framework's success also hinges on the assumption that prompt evolution is approximately linear in the learned latent space, which has not been validated across diverse model architectures and scales. The Lyapunov-style stability constraint, while effective, might be overly restrictive, potentially preventing the model from exploring optimal solutions that require a temporary increase in complexity. Consequently, although performance has been demonstrated, these are results from a single model, and the effectiveness of these simplifications (a single global operator and regularization) may diminish as conditions become more complex. \n\nLimited Experimental Scope: The paper's empirical validation is limited to a single backbone architecture (ViT-Base/16), so it remains unverified whether PAE's effectiveness generalizes to other architectures. It also does not investigate how performance varies with model scale (e.g., larger or smaller ViTs). Lastly, the absence of a direct comparative analysis with other major PEFT (Parameter-Efficient Fine-Tuning) families, such as LoRA, makes it difficult to fully assess the pros and cons of PAE-enhanced VPT within the broader PEFT landscape. An exploration of its orthogonality and complementarity with these methods would be beneficial.\n\nIn conclusion, the study has demonstrated the success of the KLD framework's simplifying assumptions within the specific context of ViT-Base/16. However, it has not been verified whether this success will hold under more complex conditions, such as with more intricate architectures, much larger-scale models, or when combined with other PEFT techniques like LoRA. In other words, a key limitation of this research is that the possibility that the success of this simplification is a coincidence within the limited experimental scope cannot be ruled out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZsSsY5UnXF", "reviewer": "Reviewer_9KV8", "summary": "Visual prompt tuning enables parameter-efficient fine-tuning. However, existing VPT variants often suffer from unstable training. To address this challenge, the authors propose the Prompt-Agnostic Evolution (PAE) by explicitly modeling the dynamics of learnable prompts.", "questions": "My question mainly focuses on the discussions on projection-based prompt tuning methods. Other than that, this paper looks good to me.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of unstable VPT training is well defined. The observations on shallow- and deep-layer prompts are interesting. The clear mismatch for gradient oscillations is something researchers might be interested in. \n\n2. The paper is easy to follow, and the problem-solving is practical.\n\n3. The ablation study is sufficient.", "weaknesses": "1. The masking then project idea sounds similar to projection-based (a.k.a instance-aware) prompt tuning [1-2], where these papers use input projection directly to guide prompt training. The authors need to discuss them and clearly separate their differences. \n\n2. The format in conclusion is a little bit weird. Please fix it.\n\n3. The motivation of this paper can be clearer, for example, why the authors want to discover frequency shortcuts. I understand the observations; however, their motivations are unclear to me.\n\n[1] Visual Instance-aware Prompt Tuning\n\n[2] All You Need is One: Capsule Prompt Tuning with a Single Vector", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o6AU19X4tR", "reviewer": "Reviewer_oUNy", "summary": "Visual cue adaptation effectively adapts frozen pre-trained models by inserting a small number of learnable cue tokens into each layer of ViT. However, existing VPT variants often suffer from dynamic instability in training, manifested by gradient oscillations, early stagnation of shallow-layer cues, and high-variance oscillations of deep-layer cues, leading to slow convergence and ultimately degraded performance. To address these challenges, this paper proposes Prompt-Agnostic Evolution (PAE), which enhances VPT by explicitly modeling the dynamic evolution of learnable cues.", "questions": "The paper simplifies the evolution of cues across layers to a global linear transformation (the Koopman operator) in a shared latent space. While this linear assumption effectively promotes inter-layer consistency and stabilizes training, does it limit the model's expressive power, especially when dealing with complex tasks or very deep networks with highly nonlinear inter-layer relationships? Exploring nonlinear dynamical models or introducing layer-wise adaptive evolutionary operators might yield further improvements in model performance at the expense of a small degree of simplicity.\n\nReframing cue tuning as a dynamical system is a novel perspective. In addition to stabilizing training and accelerating convergence, can this framework also be used to improve model interpretability? For example, by analyzing the eigenvalues ​​and eigenvectors of the learned Koopman operator, can we gain insight into how the model adjusts its internal representation to adapt to new tasks? Or can we identify the \"dynamic patterns\" that are crucial for specific tasks, thereby providing theoretical guidance for more effective cue design?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. PAE is lightweight and independent of prompts, a significant decoupling advantage.\n2. It performs well, especially on classification tasks.\n3. The derivation is robust, with no major issues.", "weaknesses": "This method introduces multiple new hyperparameters, increasing the complexity of model tuning. Ablation experiments in the paper show that model performance is sensitive to the choice of these parameters. This means that in real-world applications, tedious search and fine-tuning for different tasks may be required to achieve optimal results, which somewhat limits its plug-and-play usability.\n\nThe experimental validation in the paper focuses primarily on the specific architecture ViT-Base and the task of image classification. While achieving convincing results on the FGVC and VTAB-1k datasets, the effectiveness of this method on a wider range of visual tasks, such as object detection and semantic segmentation, has not yet been verified. This raises questions about its general applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)