Title: Visual Prompt-Agnostic Evolution
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: computer vision, visual prompt tuning

Abstract: Visual Prompt Tuning (VPT) enables effective adaptation of a frozen Vision Transformer (ViT) to downstream tasks by inserting a small number of learnable prompt tokens into the token sequence at each layer. However, we observe that existing VPT variants often suffer from unstable training dynamics, characterized by gradient oscillations. A closer layer-wise analysis reveals that shallow-layer prompts tend to stagnate early, while deeper-layer prompts exhibit high-variance oscillations, leading to a cross-layer mismatch. These issues contribute to slower convergence and degraded final performance. To address these challenges, we propose the Prompt-Agnostic Evolution (PAE) framework, which can strengthen vision prompt tuning by explicitly modeling the dynamics of learnable prompts. patterns that the backbone inherently exploits for recognition. To ensure coherent evolution across layers, we further employ a shared Koopman operator, which imposes a global linear transformation rather than uncoordinated, layer-specific updates. Finally, inspired by Lyapunov stability theory, we introduce a regularizer that constrains error amplification during evolution, leading to improved training robustness. Extensive experiments demonstrate that using PAE with VPT variants not only accelerates convergence with an average 1.48× speedup but also yields 1–3% classification accuracy gains on two image classification benchmarks, including FGVC and VTAB-1k. Beyond performance, PAE remains prompt-agnostic and lightweight, and it integrates seamlessly with diverse VPT variants without backbone modification or inference-time changes, and thus provides a practical and scalable solution for advancing prompt tuning.