Token Coordinated Prompt Attention is Needed for Visual Prompting
Authors: Zichen Liu
Xu Zou
Gang Hua
Jiahuan Zhou
Venue: ICML 2025 poster (2025)
Keywords: Visual Prompting
Prompt Learning
Primary area: deep_learning->attention_mechanisms

Abstract: Visual prompting techniques are widely used to efficiently fine-tune pretrained Vision Transformers (ViT) by learning a small set of shared prompts for all tokens. However, existing methods overlook the unique roles of different tokens in conveying discriminative information and interact with all tokens using the same prompts, thereby limiting the representational capacity of ViT. This often leads to indistinguishable and biased prompt-extracted features, hindering performance.  To address this issue, we propose a plug-and-play Token Coordinated Prompt Attention (TCPA) module, which assigns specific coordinated prompts to different tokens for attention-based interactions. Firstly, recognizing the distinct functions of CLS and image tokens-global information aggregation and local feature extraction, we disentangle the prompts into CLS Prompts and Image Prompts, which interact exclusively with CLS tokens and image tokens through attention mechanisms. This enhances their respective discriminative abilities. Furthermore, as different image tokens correspond to distinct image patches and contain diverse information, we employ a matching function to automatically assign coordinated prompts to individual tokens. This enables more precise attention interactions, improving the diversity and representational capacity of the extracted features.  Extensive experiments across various benchmarks demonstrate that TCPA significantly enhances the diversity and discriminative power of the extracted features.

Token Coordinated Prompt Attention is Needed for Visual Prompting
Zichen Liu'
Xu Zou
Gang Hua
Jiahuan Zhou
Abstract
Attention Map
ttention Map Image Tokens
CLS Tokea
Row B. Columa
Column
Colomn
Visual prompting techniques are widely used to efficiently fine-tune pretrained Vision Trans- formers (ViT) by learning a small set of shared prompts for all tokens. However, existing meth- ods overlook the unique roles of different tokens in conveying discriminative information and interact with all tokens using the same prompts, thereby limiting the representational capacity of ViT. This often leads to indistinguishable and biased prompt-extracted features, hindering performance. To address this issue, we propose plug-and-play Token Coordinated Prompt Attention (TCPA) module, which assigns specific coordinated prompts different tokens for attention-based interactions. Firstly, recognizing the distinct functions of CLS and image tokens- global information aggregation and local feature extraction, we disentangle the prompts into CLS Prompts and Image Prompts, which interact exclusively with CLS tokens and image tokens through attention mechanisms. This enhances their respective discriminative abilities. Fur- thermore, as different image tokens correspond to distinct image patches and contain diverse information, we employ a matching function to automatically assign coordinated prompts to individual tokens. This enables more precise attention interactions, improving the diversity and representational capacity of the extracted features. Extensive experiments across various benchmarks demonstrate that TCPA significantly enhances the diversity and discriminative power of extracted features. The code is available http://ggihub..co//h/ujjaaa /9CMM TCPA.
VPH
HUCA
Benchmark
VAB Benchmark
Average Ascurnty (ho)
TTPPA
TTCPASS
TTECPA
B VEPT
DAMYP
+TCPA
TTCPA
-TCPA
AnnVD
-TCPA
100
120
140
160
100
80
100
120
140
160
100
Inference Time (ms iteration)
Inference Time (ms (ttration)
Visualization of the attention map. The existing visual prompting method VPT (Jia et al., 2022) learns the same prompts for all tokens. resulting in extracted information that lacks distinguishability and comprehensiveness. Our TCPA selects corresponding prompts for different tokens and performs attention interaction, thereby enhancing the diversity and discriminability of the extracted information. Below: Comparison of time overhead and performance.
Figure 1. Above:
Introduction
In recent years, the cetrraining-finetuiinn strategy has be- come foundational paradigm in the deep learning field, significantly advancing the progress of various multi-media technologies (Jang et al., 2019; Guo et al., 2019; Iofinova et al., 2022: Xu al., 2025; & Zhou, 2025: Yao al., 2025). However, as the sizes of models and datasets have rapidly exploded, such popular paradigm has faced crit- ical challenges due to its high storage and computational costs (Jia et al., 2022). Addressing this, recent research (He et al., 2020; Cai et al., 2020; Zhang et al., 2020; Han et al., 2023) has focused on efficiently adapting pretrained models to specific downstream tasks. Among them, visual prompt-
Wangxuan Institute of Computer Technology. Peking Univer- sity, Beijing, China "School of Artificial Intelligence and Automa- tion. Huazhong University of Science and Technology, Wuhan, China ' Ammazon.com Inc, Bellevue, WA, USA. Correspondence to: Jiahuan Zhou <jiahuanzhou @ @pku.edu.cn>>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
ing has emerged a leading player (Bahng et al., 2022; Jia et al., 2022; Huang al., 2023) by introducing minimal set of learnable prompts into the latest vision transformer (ViT) without retraining the original model parameters.
kens, we first disentangle the prompts into CLS prompts and Image prompts. We then match corresponding prompts to different tokens, fostering coordinated interactions between tokens and prompts and enhancing the discriminative ability of the extracted features. (3) Extensive experiments on vari- ous benchmarks show that TCPA consistently enhances the performance of existing state-of-the-art visual prompting methods.
Existing visual prompting methods can be primarily cate- gorized into two branches. Various works involve adding learnable prompts directly to the imput sample itself, guid- ing the model to focus on discriminative information at the input-level (Bahng et al., 2022; Chen et al., 2023; Huang et al., 2023; Tsao et al., 2024). Besides, another branch introduces learnable tokens as prompts incorporated into each self-attention layer in ViT (Jia et al., 2022; Han et al., 2023; Yoo et al., 2023; Wang et al., 2024b). They aim to continuously prompt the model throughout the entire feature extraction process, facilitating the extraction of discrimina- tive features. However, these methods usually learn and leverage the same prompt for all tokens without considering the different functionalities of CLS and image tokens, as well as the varying discriminative information conveyed by different image tokens. Consequently, this leads to different tokens focusing on similar regions and extracting biased discriminative information as shown in Figure 1, thereby limiting the representation ability of VIT.
2. Related Work
2.1. Parameter-Efticient Fine- Tine-Tuning
Vision Transformer (ViT) has made significant strides in computer vision research (Dosovitskiy et al., 2020; Liu et al., 2021b; Arnab et al., 2021; Chen et al., 2021a; Wang et al., 2021). However, the continuously increasing model sizes and datasets pose challenges in fully fine-tuning pretrained ViT models for downstream tasks, leading to substantial storage and computational cost. Consequently, recent stud- ies (Zhang et al., 2020; Jia et al., 2022; Han et al., 2023) have shifted focus towards reducing the number of trainable parameters to streamline the fine- tne-ttunig process, broadly categorized as partial taning-based, extra module-based, and prompt learning- -based approaches.
To address the above issues, we introduce plug-and-play Token Coordinated Prompt Attention (TCPA) module. It assigns specific coordinated prompts to different tokens for targeted iiiention-based interactions, allowing each prompt to contribute effectively to the extraction of comprehensive and discriminative information. Specifically, considering that CLSS tokens and image tokens focus on global informa- tion aggregation and local feature extraction, respectively, we design CLS prompts and Image prompts for the CLS token and image tokens. These prompts interact exclusively with CLS tokens and image tokens within the attention blocks, thereby enhancing the discriminability of the ex- tracted features. Furthermore, since different image tokens correspond to distinet image patches and the information they need to extract varies, we further disentangle CLS prompts and Image prompts into CL-S prompt Pool and an Image prompt Pool, each composed of multiple prompts. Token-ce coordiiatdd prompts are automatically assigned to each token, improving the diversity of discriminative infor- mation in the extracted features.
Partial tuning- based methods (Yosinski et al., 2014; He et al., 2020; Noroozi & Favaro, 2016; Zhang et al., 2016) aim to retain most of the pretrained backbone while fine-tuning a smaller subset of parameters. Although straightforward and easy to implement, these methods often exhibit a notice- able performance gap compared to full fine-tuning (Chen et al., 2021b). On the other hand, extra module-based ap- proaches (Rebuffi et al., 2017; Zhang et al., 2020; Cai et al., 2020; Pfeiffer et al., 2020; Zaken et al., 2021) introduce additional learnable plug-in architectures to fine-tune the pretrained model. However, these approaches are often tai- lored to specific architectures, limiting their applicability to other models. Moreover, the introduction of additional learnable parameters poses practical challenges, making them less feasible in real-world scenarios (Jia et al., 2022; Han et al., 2023).
2.2. Prompt Learning
Prompt learning techniques initially emerged in the realm of natural language processing (NLP), involving the incor- poration of a small set of learnable soft prompts into input texts to customize language models for specific downstream tasks (Li & Liang, 2021: Liu et al., 2021a). Recent research has extended prompt learning to visual tasks, known as vi- sual prompt tuning or visual prompting (Bahng et al., 2022: Jia et al., 2022; Han et al., 2023; Liu et al., 2024c;a;b). Compared to partial tuning- and extra module-based methods, visual prompting- approaches introduce sig-
To sum up, the main contributions of this work are: (1) To address the issues in existing visual prompting methods, we introduce a glug-and-play Token Coordinated Prompt Attention (TCPA) module, which assigns specific prompts to different tokens for targeted attention breettion-bssed interactions, allowing each prompt to contribute effectively to the extrac- tion of comprehensive and discriminative information. (2) Considering the differences in the information extracted by CLS and image tokens, as well as among different image to-
2

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
Classifier
Lice
Partition
Eq.10
Eq.11
Image Promgrt
P!
Eq.9
x 4.
Add & Norm
Image Toksen
Affinity
CLS Prompr
PC
Eq.9
Eq. 10
Eq.11
[CLS]
Thansfornz Lzger
Feed Forward
Affinity
Add & Norm
ttention Map
Mask 1
Mask
[CLS)
Lines 1//1
p
P
[CL.]]
Image Token
Image
Tokin
Token-Oriented Coordinated Prompt ttention
(CLS)
Prompt
Image Token
Embedding
Prompl
AtAt money
A Allention Map Attn'
Image Tokan
Input Image
(a) Overall Pipeline
(b) Token Coordinated Prompt A ttention
Figure 2. The overall pipeline of proposed TCPA. For each input sample, embeddings for each image patch first obtained through the embedding layer. Then, CLS and image tokens adaptively select appropriate prompts from the corresponding CLS and Image Prompt Pools and generate binary mask. This binary mask is then fed into the attention module mask certain values in the attention map, enabling attention-base based interactions between different tokens and different prompts.
nificantly fewer additional parameters and exhibit superior compatibility with models of various architectures (Jia et al., 2022; Han et al., 2023).
they the same prompts for all tokens without consid- ering the distinct roles of CLS and image tokens, well as the differences in discriminative information extracted by various image tokens. This results in the features ex- tracted by different tokens being neither distinguishable nor comprehensive, which limits the model's performance.
Specifically, existing visual prompting methods can mainly categorized into two types based on the location where prompts are applied: those added the input im- age and those added within the token sequence. Prompting methods added on the input image overlay learnable visual prompts onto the original image to adjust pretrained models from the input level, enabling them to adapt to downstream tasks (Bahng et al., 2022; Huang et al., 2023; Chen et al., 2023; Wang et al., 2023; Tsao et al., 2024). These methods adjust pretrained models only in the input image level, mak- ing them adaptable different network structures. However, since visual prompts are not used in the middle layers of the network, the representation capacity of prompts in such methods is limited, resulting in limited performance.
3. Token Coordinated Prompt A Attention
In this section, we illustrate the proposed TCPA in detail, and the overall pipeline is depicted in Figure 2.
3.1. Notations
In the architecture of a pretrained Vision Transformer (ViT) backbone, denoted as M, there are instances of MSA (Multi-Head Sclf-Attention) blocks, symbolized as (B, } Each block, B,, integrates multi-head self-attention with networks, incorporating both LayerNorm and residual pathways for enhanced processing. When pro- cessing input image with dimensions IR XXXC this image is partitioned into N patches of uniform size {æ, E IR R }= Here, (H, W) represents the size of x, C is the channel count, and (h, w) denotes the size of each patch æ;, The transformation of each patch æ; into a D-dimensional feature space is given by:
Another category of visual prompting involves introducing learnable tokens into the intermediate layers the model, which undergos self-attention along with CLS and image tokens, thereby extracting discriminative features (Jia et al., 2022; Han et al., 2023; Yoo et al., 2023; Wang et al., 2024a). For instance, VPT (Jia et al., 2022) introduces learnable tokens at every layer of the vision transformer, allowing for individual adjustments across layers to better suit down- stream tasks. These methods continuously provide prompts during the feature extraction process of the model. However,
h E(w:))
(1)
3

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
where h RR, and E(-) is the embedding layer of M. These embedded patches, {h: } along with classifi- cation (CLS) token C₁ E IR are sequentially processed through the L MSA blocks {B, 3 The operation can be summarized as follows:
Similarly, we input the image prompt pj along with the CLS token Gj and image tokens (h² hxy) into the MSA
block By, obtaining the output for the image tokens:
id [c] (+1 1+t1,P+t110 h
+++ (, 14
NxD:
(7)
[9)+1, 1
hi, ,.'D.
(2)
where the subscript d in o 1: 3.d indicates that these outputs will be discarded. Through the equation above, we obtain the output for the image tokens (hi*1 .... h, ) which, together with the previously obtained output of the CLS token Cj+1, serves as the input for the next MSA block Bj+1-
where the "[]" denotes the concatenation of the vectors. The final classification is conducted by passing the last MSA block's output, CL+1. through a classifier Hi
y = =((CL+1)
(3)
3.3. Coordinated Prompt Attention of Different Image Tokens
3.2. Coordinated Prompt Attention of CLS and Image Tokens
In the previous text, we disentangle prompts into CLS and image prompts based on their distinct roles. However, in vision transformers, different image tokens correspond to different image patches with varying discriminative infor- mation. Using the same prompts for all image tokens can make the extracted features indistinguishable and biased. Thus, we further introduce coordinated prompt attention of different image tokens to enhance the pretrained model's ability to extract rich, discriminative information from the input image.
In the vision transformer, an image æ is initially divided into numerous small patches, which are then converted into corresponding image tokens via an embedding layer. Dur- ing the attention process, image tokens continuously extract discriminative information from the input sample æ. This in- formation is subsequently aggregated through CLS token, summarizing the insights gathered from all image tokens for final classification. It is evident that the role of image tokens to extract discriminative information, whereas the CLS token's 's purpose is aggregate this information and facilitate classification, highlighting the distinct functions of these two types of tokens. Hence, we design Coordinated Prompt Attention of CLS and Image Tokens, which disen- tangles prompts for CLS and image tokens, aiding them in better fulfilling their respective functions.
To simplify notation, in the following discussion, we will not explicitly differentiate prompts from different layers. It's important note that while the parameters of prompts vary across layers, the processing method for prompts re- mains consistent throughout. Specifically, we disentangle the image prompt p into a image prompt pool Pi composed of multiple image prompts:
Specifically, we disentangle prompts for CLS and image tokens, denoted as De and p', respectively:
p =pp}}==
(4)
p' (ppk )=)
(8)
P' = (p) 1/
(5)
where Hi represents the learnable indicator corresponding to the prompt pk. used for selecting the prompt based on the image token.
where pf E RL and p E RLoxD are CLS and image prompt for j-th MSA block By.
For each image token hi the distance between hy and the prompt indicator ki can be measured via cosine distance S(,,))
Then we feed the CLS token сj. image tokens (hi.... ,,²y), and the CLS prompt pf together into the
MSA block By, obtaining the corresponding output:
S(h k()) = - cos Koj))
[cj+1. P44
Pj+1: h 2+1d
h
(9)
([, pf. ny.... h2),,
(6)
= 1
|
add where the subscript d in p h / that these outputs will be discarded and not utilized by subsequent MSA layers. In the equation above, only the output CLS token continues to be used, and therefore, the CLS prompt only affects the CLS token, not the image tokens.
i+1dd N
++1.d indicates
Through the above process, obtain the affinity matrix A E NNN. between image tokens and different image prompts, where Am,k = S(h,,, +c)) Then, we perform binarization on the matrix A, setting the top K; largest elements in each row to 1 while assigning 0 to all other ele- ments. Specifically, the binarized matrix A E (0, 1] NxN.
4

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
is defined as follows:
weights of all tokens and prompts, Attn = Softmax Then, based on the token- -oonnpppppt matching, we generate a binary mask matrix M and compute Attn M V. This ap- proach calculates the attention weights only once, enabling efficient interaction between different tokens and prompts.
QK
vdz
Ki Σ 1(k==Ta (m)) ()))=)(())>0))
mm,k k
(10)
where T(()) denotes the indicator function, which takes the value of if the condition inside holds and 0 otherwise; Tm represents the index sequence obtained by sorting the elements of the i-th row of matrix A in descending order, i.e., satisfying Am (m, 2 A Am, 2 ... 2 A Am, (n); and (s) corresponds to the column index of the s-th largest element after sorting. Through this operation, we obtain a binary mask matrix M, which selects the top K; largest elements in each row of A while suppressing the influence of other irrelevant elements. Then, we align the dimensions of Å with the dimensions of the attention map to obtain the final image token mask:
Agmrinn Wright
A.30
1:15
1.10
Thkes Joten (Qwry))
lenge Tokay
goioo budee
(Fay
30m)
(4) 3D Allestion Map of VPT
(b) 2D Attention Map of VPT
Amredon Weigh
8.04
0, - Am--1k otherwise
m
M
(11)
Preege
Tates fuder (Gwery)
bokes (Bir)
Ty
(c) 3D Attenionn Map of Ours
(4) 2D Amention Map of Ours
To guide the CLS token corresponding to different sam- ples in better aggregating global information, we also dis- entangle the CLS prompt pd into CLS prompt pool k=1 In a similar manner to the image
Figure 3. 3D and 2D attention map of existing visual prompting method VPT (Jia al., 2022) and Ours.
pe = {(p). k=) tokens, we can obtain the affinity matrix between the CLS token and the CLS prompts. Then, by further binarizing and expanding the dimensions, we obtain the mask M corresponding to the CLS token.
3.4. Overall Optimization
As mentioned above, our TCPA introduces only few addi- tional parameters: CLS prompt pool P and image prompt pool Pt, Following (Jia et al., 2022), during training, we maintain the pretrained model's encoder frozen while al- lowing only the classification head to be trainable. We denote all learnable parameters as = {P", pt, H}. The optimization objective is as follows:
In Vision Transformers, the core operation of the attention module is:
QKT
Attn Softmax
(12)
Vix
where Attn is the attention map. We concatenate the two masks, M and M corresponding to the CLS token and the image token, then expand them to the same dimensions as the attention map Attn, resulting in the final mask M. Finally, perform element- wise multiplication between Attn and the mask M to obtain the updated attention map for subsequent operations:
arg min Lee (y. +A: Σ (hm ) Σ S(c,, Kg), +
(14)
where Loe is cross-entropy loss, yet is the label of image æ, x; and de are weighting parameters.
Attn's Attn O M,
(13)
4. Discussion and Analysis
where 3 denotes the element-wise multiplication.
To further analyze and validate the effectiveness of our method, this section provides mathematical and experimen- tal analysis of why existing visual prompting methods ex- tract discriminative information that is singular and insuf- ficient, while our TCPA method extracts comprehensive discriminative information.
Although our TCPA selects specific prompts for each token, the additional computational overhead is limited only to the calculation of cosine distance for prompt selection and the self-atten self-attention process, with no increase in the computation of the feed-forward network. Furthermore, we can achieve the effect of multiple groups of tokens undergoing attention separately through a single attention computation by uti- lizing token masking. In TCPA, we compute the attention
Theorem 4.1. Self-attention is low rank. (Proved in (Wang et al., 2020)). Let A E x=xx be a self attenttttnno matrix, and e RR be a column vector of value matrix V. Then, there
5

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
Table The comparison results on HTA benchmark. Partial, Extra, and Prompting represent partial tuning-basee extra module- module-based.
and prompt learning based methods respectively.
Methods
DTD
CUB
Bird
Dog
Flower
Food
Cifar 100
CitarIo
GTSRB
SVHN
Avg
Full
64.3
87.3
82.7
89.4
98.8
84.9
68.9
97.4
97.1
87,4
85.8
Linear
63.2
85.3
75.9
86.2
97.9
84.4
63.4
96.3
68.0
36.6
75.7
Pornial
Partial
NowrP5514 14
70.1
85.6
77.8
85.5
98.2
83.8
78.0
95.0
89.3
82.4
84.6
MI.P
CVPR
66.2
85.1
77.3
84.9
97.9
84.6
77.5
93.2
71.8
60.5
79.9
Bias
17
69.8
88.4
84.2
91.2
98.8
86.2
82.9
96.9
89.9
82.5
87.1
Extra
Sidetune
ECCV'20
57.7
84.7
75.8
85.8
96.9
78.7
68.8
90.4
90.9
80.5
81.0
Adapter
N0
62.7
87.1
84.3
89.8
98.5
86.0
74.2
97.7
91.1
36.3
80.8
Former
64.2
87.3
84.1
88.1
98.4
85.7
79.4
96.5
91.7
83.0
85.8
ETVPT
ICCV23
66.8
88.4
84.2
91.3
99.0
84.0
80.4
97.1
91.0
79.2
86.1
LION
AAA/24
83.6
90.5
65.4
90.8
VP
arXiv'22
59.5
84.6
77.7
84.5
97.7
80.5
78.7
94.2
89.4
87.6
83.4
+TCPA
This Poper
62.3(428)
86.7(=2.1)
78.7(-10)
88.6(-3.1
99.3(-1.6)
81.9(-1.4)
81.4(+2.7)
95. (44.9)
90.7(-13)
89.6(-2.))
85.4(-20)
Prompting
VPT
ECCV'22
65.8
88.5
84.2
90.2
99.0
83.3
78.8
96.8
90.7
78.1
85.5
+TCPA
This Puper
67.6(-18)
90.7(222)
85.5
91.7(+1.5)
(9.2
84.8
(9.9(1..))
(8.9
()
79.4..1.)
87.0(-1.5)
DAMVP
CVPR
73.1
87.5
82.1
92.3
99.2
86.9
88.1
97.3
90.6
87.9
88.5
+TCPA
This Puper
74.3
88.9 1.)
82.7
93.6
(9.3
88.7(+1.))
89.7(.1.6)
97.8.a0.5)
93.8(43.2) 2)
90.4.2.5)
(9.9(-1.4)
AutoVP
1CLR'24
62.5
85.4
83,5
90,4
82.3
77.9
95.2
93.1
92,9
85.4
+TCPA
This Poper
65.0
88.0(-26)
85.6
92.5
-1.2
33.6
79.3++
97.4(+2.)
96,2(+3.1)
(3.5
87.2
VFPT
24
69,9
88.1
82,8
89.5
98.4
88.7
90.4
97.4
92.1
94,4
89.2
+TCPA
This Poper
71.2(-133)
89.5(-14)
83.6.-88)
91.5x+2.))
(9.2()n.)
89.2(.0.5) 91.6(+1.2 98.4(+14)
94.2(42.1
95.6(-1.3)
90.4(-12)
visual prompting method VPT in both 3D and 2D. As shown in Figure. 3, the attention regions of prompts in conventional visual prompting methods are highly similar, leading to CLS and image tokens extracting nearly identical features.
Table The comparison results VTAB benchmark. Utilize the ViT-B/16 pretrained with supervised training on ImageNet-21k as the backbone.
Methods
Natural
Specialized
Structured
Full
75.9
83.4
47.6
In contrast, our proposed TCPA module enhances more diverse attention across prompts, CLS tokens, and image tokens. This is because our method selects different prompts for different tokens and performs attention-basee interac- tions, thereby encouraging the model to extract more diverse and comprehensive discriminative information.
VP
77.3
80.1
53.8
+TCPA
(8.1(408)
82.6(+2.5)
55,4(+1.)
VPT
78.5
82.4
55.0
+TCPA
79.7(112)
84.3(419)
56,2(112)
DAMVP
79.1
83.4
56.2
+TCPA
80.4.1.)
85.5(42.1)
57.1
AutoVP
78.4
83.1
55.8
+ +TCPА
79.3(+0.)
85.2(+2.))
56.9(+1.D)
5. Experiments
exists a low-rank matrix À E Raax satisfying
5.1. Datasets
Pr( - AuF'll < AlI 1-0(1,
(15)
Building upon (Jia al., 2022: Huang al., 2023), the ex- periments are conducted on HTA (Huang et al., 2023) bench- mark, including: DTD (Cimpoi et al., 2014), CUB-200- 2011 (Wah et al., 201 1), NABirds (Horn et al., 2015), Stan- ford Dogs (Khosla et al., 201 1), Oxford Flowers (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), Ci- 100 (Krizhevsky al., 2009). Cifar10 (Krizhevsky et al., 2009). GTSRB (Stallkamp et al., 2012), and SVHN (Netzer et al., 2011).
where the rank of Å is bounded, i.e., rank(A) (log(nn)).
Theorem 4.2. Self-attention low-rank after prompting. (Proved in (Kim al., 2024)). For any low-rank matri- ces An E TRxn and Antra E (m++m) (n+m) satisfying Pr(lIAu" -Ao < Av
o(1), we have
rank(An rank AA.) ) (l)))mm)),
(16)
Moreover, following (Jia et al., 2022; Han et al., 2023), more experiments are conducted on the VTAB benchmark (Zhai et al., 2019) which includes 19 visual tasks. These tasks are categorized into three groups: Natural, for routine image recognition; Specialized, for domain speciffc applications such as medical imaging; and Structured, for the analysis of intricate scenes, like 3D object recognition.
where is the number of prompts.
Through the above two theorems, we can see that the self- attention matrix in existing prompt learning methods is low- rank. This indicates that different prompts in these methods tend to focus on the same image regions. To further demon- strate this, we visualize the attention maps of the existing
6

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
5.2. Comparison Methods
Table 3. The influence of components in TCPA. " "V" represent with this component. R-TCPA represents the coordinated prompt atten- tion of CLS and image tokens. T-TCPA represents the coordinated prompt attention of different image tokens.
We compare TCPA with the parameter-efficient fine-tuning and visual prompting methods. We also report the fully- tuning results as bascline. Specifically, the parameter- efficient fine-tuning methods include the partial tuning- based models (Linear (Iofinova et al., 2022), Partial (Yosin- ski et al., 2014), MLP (He et al., 2020)) and the ex- tra module-based ones (Sidetune (Rebuffi et al., 2017), Bias (Zhang et al., 2020), Adapter (Cai et al., 2020), Adapt- Former (Chen al., 2022)). For visual prompting methods, various latest visual prompting methods such as VP (Bahng et al., 2022), VPT (Jia et al., 2022), DAMVP (Huang et al., 2023), Yoo et al (Yoo et al., 2023), VPT (Han et al., 2023), LION (Wang et al., 2023), Au (Tsao et al., 2024) and VFPT (Zeng et al., 2024) are evaluated.
Components
Datasets
R-TCPA
T-TCPA
CUB
Dog
GTSRB
88.1
89.5
92.1
88.9
90.2
93.0
/
/
89.5
91.5
94.1
To further validate the effectiveness of our TCPA, follow- ing (Jia et al., 2022; Han et al., 2023), we also conduct experiments on VTAB (Zhai et al., 2019) benchmark. As shown in Table 2. compared to VPT, the integration of TCPA results in performance improvements of 1.2%, 1.9%. and 1.2% across downstream tasks in groups Natural, Special- ized, and Structured, respectively. Moreover, TCPA con- sistently enhances performance based VP, DAMVP, and VP. Specifically, VP+TCPA shows improvements of 0.8%, 2.5%, and 1.6%, DAMVP+TCPA shows improve- ments of 1.3%, 2. %, and 0.9%, and AutoVP+TCPA also exhibits performance enhancements of 0.9%, 2.1%, and 1.1% across the same groups of downstream tasks. This further demonstrates TCPA's robustness across a variety of downstream tasks.
5.3. Implementation Details
To fully validate the effectiveness of our proposed TCPA, we implement TCPA based on several representative visual prompting methods from recent years. For the token-level methods VPT (Jia et al., 2022) and VFPT (Zeng et al., 2024), replace their learnable prompt tokens with our TCPA. For the input-level prompting methods VP (Bahng et al., 2022), DAMVP (Huang et al., 2023), and Vu (Tsao et al., 2024), while retaining their prompts added to the input images, we introduce our TCPA at the token level. The ViT- B/16 (Dosovitskiy et al., 2020) supervised by ImageNet- 21k (Deng et al., 2009) is used as the backbone. Following DAMVP (Huang et al., 2023), we train for 100 epochs all datasets. The AdamW (Loshchilov & Hutter, 2017) optimizer and cosine annealing are used for optimization. The weighting parameters A and A are set to 0.5. The length of the size of CLS prompt pool Ne and size of image prompt pool N; are set to 10 and 20 respectively.
5.5. Ablation
5.5.1. INFLUENCE OF DIFFERENT COMPONENTS
To further validate the effectiveness of each component we proposed, we conduct ablation studies on the two main com- ponents of TCPA: R-TCPA and T-TCPA. When none of the modules are utilized, the method degenerates to the original VPT approach. Conversely, employing all three modules constitutes the complete VPT+TCPA method. As shown in Table 3, the introduction of the R-TCPA module leads to a performance increase of 0.8%-0.9% This improvement is attributed R-TCPA's disentanglement of CLS and im- age prompts based on their distinct roles, guiding CLS and image tokens to fulfill different functions. Further incorpo- rating T-TCPA allows different image tokens to adaptively select suitable prompts from the prompt pool, thoroughly capturing diverse discriminative information from the input sample, hence boosting model performance by an additional 0.6%--.11%
5.4. Comparison with State-of-the-arts
We first conduct experiments on HTA using the IeageNet- 21k supervised ViT-B/16 (Dosovitskiy et al., 2020) as the pretrained model. As shown in Table 1, after integrating TCPA, in comparison to DAMVP, DAMVP+TCPA shows average improvements of 1.4% across all ten datasets. Simi- lar enhancements are also observed when applied to other methods: VP+TCPA shows an average increase of 0.9%- 2.8% across the ten datasets, VPT+TCPA improved by 0.2%, AutoVP+TCPA improved by 0.6%-3. 1%, and VFPT+TCPA also improved by 0.0%. This can be primarily attributed to TCPA's explicit disentanglement of prompts based on the distinct roles of CLS and image tokens and their difference in the attention mechanism, allowing for more thorough learning of downstream task knowledge and facilitating comprehensive extraction of discriminative information, thereby boosting model performance.
5.5.2. COMPUTATIONAL COST
To validate the efficiency of our proposed method, we con- duct a computational cost analysis to compare our proposed method, TCPA, with existing visual prompting approaches. As shown in Table 4, TCPA introduces only a minimal additional time cost while enhancing model performance.
7

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
Table Comparison of training time on CUB (scconds/epoch)
with state-of art
Methods
Training Time
VP (Bahng et al., 2022)
1.94
+TCPA
2.24
VPT (Jia et al., 2022)
2.36
+TCPA
2.41
(a) DA
(b) TCPA (Ours) On
DAMVP (Huang et al., 2023)
2.14
+ +TCPA
2.27
VFPT (Zeng et al., 2024)
2.58
Figure 5. Feature 1-SNE (Van der Maaten & Hinton, 2008) visu- alization results for our proposed TCPA and comparison method DAMVP (Huang et al., 2023) on GTSRB.
+TCPA
2.65
90
88
88
can lead to overfitting, which also degrades model perfor- mance. Optimal performance is achieved when the size of the prompt pool is moderate. Notably, the image prompt pool requires a larger size than the CLS prompt pool be- cause image tokens exhibit greater variability compared to the CLS tokens used directly for classification, necessitating a broader range of prompts.
Accuncy
46
86-
84
84
82
82
10
20
50
10
30
30
Size Prompt N.
(b) Size Imagee Prompt Pool
Figure 4. Influence of hyper-parameters (size of CLS prompt pool Ne, size of image prompt pool N.) of TCPA on CUB.
5.5.4. THE T-SNE VISUALIZATION OF EXTRACTED FEATURES
To further validate the effectiveness of our method, Figure 5 presents a t-SNE (Van der Maaten & Hinton, 2008) visual- ization of features obtained by TCPA and DAMVP (Huang et al., 2023). As shown in Figure 5, we visualize the fea- tures obtained by our proposed TCPA and DAMVP via t-SNE (Van der Maaten Hinton, 2008). From the visual- ization results, it can be observed that the features extracted by DAMVP from samples of the same category are relatively scattered, and some are mixed with features from other cat- egories. In contrast, features extracted by our TCPA from the same category are tightly clustered and display clear distinctiveness from features of other categories. This is attributed to our proposed token coordinated prompt atten- tion, which can recognize more diverse and comprehensive discriminative characteristics of input images.
Despite disentangling the prompts used for the CLS and image tokens, as well as between different image tokens, the increase in computational demand is negligible. The additional computational load from the prompt matching process in TCPA is primarily achieved through vector mul- tiplication, making the increase in computational demand minimal. Although different prompts are used for differ- ent tokens, in the implementation, we input all prompts simultancously into the attention mechanism for query-key computations. During the final computation with the at- tention mechanism's 's values, different masks are applied to different tokens to enable targeted attention interactions be- tween specific tokens and prompts. This implementation strategy significantly reduces the computational overhead of our method.
6. Conclusion
5.5.3. INFLUENCE OF PAPEER-RR
In this paper, we propose a novel and-play Token Coor- dinated Prompt Attention (TCPA) module to enhance visual prompting for Vision Transformers. Unlike existing meth- ods that learn the same prompts for all tokens, TCPA dis- entangles and adaptively assigns prompts different CLS and image tokens based their distinet roles, thereby im- proving feature diversity and discriminability. Specifically, we introduce CLS Prompts and Image Prompts to inter- act exclusively with CLS and image tokens, respectively, strengthening their individual representational capacities. Furthermore, TCPA leverages a matching function to dy-
As illustrated in Figure 4, we conducted ablation experi- ments on two hyperparameters introduced in TCPA: the size of the CLS prompt pool No and the image prompt pool N;. When the size of the prompt pool is small, there is lower diversity among prompts, leading to a higher overlap of prompts selected by different tokens. This results in the features extracted from different tokens becoming indistin- guishable. Conversely, an excessive number of prompts in the pool increases the number of learnable parameters. Given the limited data in downstream tasks, this scenario
8

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
namically allocate coordinated prompts image tokens, enabling more precise and targeted attention interactions. By incorporating these mechanisms, TCPA effectively miti- gates the limitations of conventional visual prompting, lead- ing to richer, more diverse feature extraction and improved model performance, as demonstrated by experimental and visualization results.
on Computer Vision (ICCV), pp. 347-356, 2021a. doi: 10.1 N0.10999CC4449222221100000 200400
Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transform- ers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 1566441666678, 2022.
Chen, X., Xie, S., and He, K. An empirical study of training self-supervised vision transformers. In 2021 TEEE/CVE International Conference on Computer Vision (ICCV), pp. 9620-9629, 2021b. doi: 10.11099/CCV44992222221 00950.
Acknowledgments
This work was supported by the National Key R&D Pro- gram of China 2024YFA14100000 and the National Natural Science Foundation of China (62376011).
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014. doi: 10.100//VVP2220444666
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
References
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x 16 words:
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučié, M., and Schmid, C. Vivit: A video vision transformer. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2816-6826, 2021. doi: 10.1199/CCV448922. 2021.00676
Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Guo, Y., Shi, H., Kumar, A., Grauman, K., Rosing, T., and Feris, R. Spottune: Transfer learning through adaptive fine-tuning. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4800 4809, 2019. doi: 10.1 N0.1109/CVPR.2019.000
Bahng. H., Jahanian, A., Sankaranarayanan, S., and Isola, P. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.I 17274, 2022.
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101- mining discriminative components with random forests. In ECCV, pp. 446-461, Cham, 2014. Springer Interna- tional Publishing.
Han, C., Wang, Q., Cui, Y., Cao, Z. Wang, W., Qi. S., and Liu, D. E'ypt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.13770 2023.
Cai, H., Gan, C., Massachusetts Institute of Technology, L. Z., and Massachusetts Institute of Technology, S. H. Tinytl: reduce memory, not parameters for efficient on- device learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713888 178171389954..
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735. 2020. doi: 10.1 122119//CVR42200.200000000 202.22200000020099955
Chen, A., Yao, Y., Chen, P.-Y., Zhang, Y., and Liu, S. Un- derstanding and improving visual prompting: A label- mapping perspective. In 2023 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp- 19133-19143, 2023. doi: 10.1109/CVPRS27292200 01834.
Horn, G. V., Branson, S., Farrell, R., Haber, S., Barry, J.. Ipeirotis, P., Perona, P., and Belongie, S. J. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-graines dataset collection. In CVPR, pp. 595-604. IEEE Computer Society, 2015.
Huang, Q., Dong, X., Chen, D., Zhang, W., Wang, F., Hua, G., and Yu, N. Diversity aware meta visual prompting. 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11878-10887, 2023.
Chen, C.-F. R., Fan, Q., and Panda, R. Crossvit: Cross- attention multi-scale vision transformer for image clas- sification. In 202/ JEEE/CVF International Conference

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
Iofinova, E., Peste, A., Kurtz, M., and Alistarh, D. How well do sparse imagenet models transfer? In 2022 IEEE/CVF Conference Computer Vision and Pat- tern Recognition (CVPR), pp. 12256-12266, 2022. doi: 10.1 109/CCPS52688 200.PP505222882222000121 195.
Liu, Z.,, Sun, H., Peng, Y., and Zhou, J. Dart: Dual-modal adaptive online prompting and knowledge retention for test-time adaptation. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pp. 14106- 14114, 2024c.
Jang, Y., Lee, H., Hwang, S. J., and Shin, J. Learning what and where to transfer. In ICML. PMLR, 05 2019.
Loshchilov, and Hutter, Decoupled weight decay regu- larization, arXiv preprint arXiv: 2017.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In Computer Vision - ECCV 2022, pp. 709-727, Cham, 2022. Springer Nature Switzerland.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. 2011.
Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graph- ics and Image Processing, pp. 722-729, 2008. doi: 10. N..119/CCVGPP.200..47
Khosla, A., Jayadevaprakash, N., Yao, B., and Li, F.F.. Novel dataset for fine-grained image categorization: Stan- ford dogs. In CVPRW, 2011.
Kim, Y., Li, Y., Moitra, A., Yin, R., and Panda, P. Do we really need a large number of visual prompts? Neural Networks, 177:106390, 2024.
Noroozi, M. and Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles. In Computer Vision - ECCV 2016, pp. 69-84, Cham, 2016. Springer International Publishing.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Pfeiffer, J.. Rücklé, A., Poth, C., Kamath, A., Vulić, I, Ruder, S., Cho, K., and Gurevych, I. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779 pp. 46-54, 01 2020. doi: 10.18653/ v 1112020.emmlp-dee
Li, Q. and Zhou, J. Caprompt: Cyelic prompt aggregation for pre-trained model based class incremental learning. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 39, pp. 18421-18429, 2025.
Rebuffi, S.-A., Bilen, H.,, and Vedaldi, A. Learning multiple visual domains with residual adapters. In Proceedings of the 31st International Conference on Neural Informa- tion Processing Systems, NIPS' 17, pp. 506-516, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964..
Li, L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.001199 2021.
Liu, X., Ji, K., Fu, Y., Tam, W. L., Du, Z., Yang, Z., and Tang, J. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint ar 10.0222 2021a.
Stallkamp. J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning algo- rithms for traffic sign recognition. Neural networks, 32: 323-332, 2012. ISSN 0893-6080. doi: https://doi.oon ortg:///oooor//1... 1016/).neumet22012.2...6666
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992-10002, Los Alamitos, CA, USA, oct 2021b. IEFE Computer Society, doi: 10.10990CCV499222221000099 URL https: /ttp:///oiii/// http://lo...eeeecoopptttrrroooeo 2tc/10.1109/Iccc4433 http://////222222...000..
Tsao, H.-A., Hsiung, L., Chen, P.-Y., Liu, S., and Ho, T.-Y. AutoVP: An Automated Visual Prompting Framework and Benchmark. In The Twelfth International Conference on Learning Representations, 2024.
Van der Maaten, and Hinton. G. Visualizing data using t-sne. Journal of machine learning research, 2008.
Liu, Z., Peng, Y., and Zhou, J. Compositional prompting for anti-forgetting in domain incremental learning. Inter- national Journal of Computer Vision, pp. 1-18, 2024a.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-200 dataset. 2011.
Liu, Z., Peng, Y., and Zhou, J. Ins Efficient instance visual prompting from image itself. In ACM Multimedia 2024, 2024b. URL https: / http///ppeeeeeeeec///// net/ Forum? d=OTjo jol g8rlI.
Wang, H., Chang, J., Luo, x., Sun, J., Lin, Z., and Tian, Q. Lion: Implicit vision prompt tuning. arXiv preprint wwXiv:2303.09992. 2023.
10

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint 20xi::20060047666. 2020.
Zhang, J. O., Sax, A., Zamir, A., Guibas, L., and Malik, J. Side-tuning: A baseline for network adaptation via additive side networks. In Computer Vision ECCV 2020, pp. 698-614 Cham, 2020. Springer International Publishing.
Wang, W., Xie, E.,, Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. Pyramid vision transformer: A versatile backbone for dense prediction without con- volutions. In 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10.1 N41109C/CC448922.2000 1.00611.
Zhang, R., Isola, P., and Efros, A. A. Colorful image col- orization. In Computer Vision EEC-EECV22100 pp. 649-666, Cham, 2016. Springer International Publishing.
2021. doi:
Wang, W., Sun, Y. Li, W., and Yang, Y. Transhp: Image classification with hierarchical prompting. Advances in Neural Information Processing Systems, 36, 2024a.
Wang, Y., Cheng, L., Fang, C., Zhang, D., Duan, M., and Wang, M. Revisiting the power of prompt for visual tuning. In Forty-first International Conference Ma- chine Learning. 2024b. URL https: http///peerrre....eeeeeee....... net/ / (eerumiid=yY9PPe-
Xu, K., Jiang, C., Xiong, P., Peng, Y., and Zhou, J. Dask: Distribution rehearsing via adaptive style kernel learning for exemplar-fie free lifelong person re-identification. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pp. 8915-8923, 2025.
Yao, Y., Liu, Z., Cui, Z., Peng. Y., and Zhou, Selective visual prompting in vision mamba. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 22083-22091 2025.
Yoo, S., Kim, E., Jung, D., Lee, J., and Yoon, S. Improving visual prompt tuning for self-supervised vision transform- ers. arXiv preprint arXiv:2306. 05067, 2023.
Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? In Pro- ceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS 14, pp. 3320 3328, Cambridge, MA, USA, 2014. MIT Press.
Zaken, B., Ravfogel, S., and Goldberg, Y. Bitfit: Simple parameter-efficient fine-tuning for transformer- arXiv preprint
based masked language-modeks arXiv:2106.10199 2021.
Zeng, R., Han, C., Wang, Q., Wu, C., Geng, T., Huang, L., Wu, N., and Liu, D. Visual fourier prompt tuning. arXiv preprint arX 2024.
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neu- mann, M., Dosovitskiy, A., et al. large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 04867, 2019.

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
A. More t-SNE Visualization Results of Extracted Features
To further validate the effectiveness of our method, we also visualized the features extracted on several other datasets using t-SNE (Van der Maaten & Hinton, 2008). As shown in Figure 6, compared the existing method DAMVP, the visualization results of our proposed TCPA show tighter clustering of samples within the same category and better separability between different categories. This is due the hierarchically disentangled visual prompting in TCPA, which disentangles prompts based on different roles and functions. Each prompt is tailored to effectively and comprehensively extract semantic information from the image samples, enhancing the discriminative ability of the extracted feature.
(a) DAM-VPon
(b) DA Cifar100
(c) SN
(d) TCPA (Ouis) CUB
(e) TCPA (Ours) Cifar100
(f) TCPA (Ours) on SVHN
Figure 6. Feature t-SNE (Van der Maaten & Hinton, 2008) visualization results for our proposed TCPA and comparison method DAMVP on CUB, Cifar100 SVHN.
B. More Attention Visualization Results
In Figure we provide attention map visualizations for additional samples. shown, existing methods, which learn the same prompt for all tokens, result in different tokens are indistinguishable and biased. In contrast, our proposed TCPA disentangles the prompts used for different tokens, thereby enhancing the diversity and discriminative ability of the features extracted by each token.
12

---

Token Coordinated Prompt Attention is Needed for Visual Prompting
Attention Map of CLS Token
Attention Map of CLS Token
Attantion Image Tokens
Attention Map Image Tokens
10
VPT
#TCPA
VPT
00
#TCPA
Figure The attention map visualization results of CL.S and image tokens from the existing visual prompting method VPT (Jia al., 2022) and our TCPA are presented.
13


Reviews:
[{"review_id": "w3HKsP1Zf4", "reviewer": "Reviewer_o6fY", "summary": "This paper introduces a token-wise prompt termed as TCPA to enrich discriminative information of tokens by assigning specific prompts to each different tokens. As a plug-and-play strategy，TCPA can be seamlessly integrate with existing prompt based methods. Experiments show that TCPA can achieve consistent performance gains across diverse benchmarks.\n\n## update after rebuttal \nThe rebuttal has well addressed my concerns, so I changed my scores from 2->3. I recommend acceptance of this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our **clearity**, **motivation** and **sound experiments**.\n\n(The images mentioned below are available at the anonymous link: https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E.)\n### Q1: Load Balance of Different Prompts\n1. Since the number of CLS and image tokens varies across different semantics, some tokens share similar meanings, while others are more distinct. As a result, prompts associated with different semantics should be selected at different frequencies. Therefore, we did not impose a strict load-balancing design for different prompts.\n2. To ensure that every prompt is optimized rather than some prompts never being selected, we *randomly choose prompts during the first 10 epochs of model training*. In the subsequent 90 epochs, prompt selection is guided by a prompt indicator, which adjusts selection based on semantic information. This design ensures that all prompts and their corresponding indicators are optimized while maintaining differing selection frequencies for prompts associated with different semantics. \n3. Furthermore, we visualize the selection frequency of prompts, as shown in Fig.1.1 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/1.1.png). The results indicate that different CLS prompts are selected at frequencies **ranging from 0.13 to 0.45**, while different image prompts are selected at frequencies **ranging from 0.05 to 0.24**. This verifies that all prompts in our method are effectively utilized, with none remaining unused.\n### Q2: Eq.14\n1. We have carefully reviewed Eq.14 and the related Eq.9 and confirmed that our formulations are **correct**.  \n2. Eq.9, $\\mathcal{S}(\\boldsymbol{h}_m^{j},\\boldsymbol{\\kappa}^i_k)=1-\\mathrm{cos}(\\boldsymbol{h}_m^{j}, \\boldsymbol{\\kappa}^i_k)$, computes the **cosine distance** between $\\boldsymbol{h}_m^{j}$ and $\\boldsymbol{\\kappa}^i_k$. A *higher* similarity between $\\boldsymbol{h}_m^{j}$ and $\\boldsymbol{\\kappa}^i_k$ results in a *larger* $\\mathrm{cos}(\\boldsymbol{h}_m^{j},\\boldsymbol{\\kappa}^i_k)$, leading to a *smaller* cosine distance $\\mathcal{S}(\\boldsymbol{h}_m^{j},\\boldsymbol{\\kappa}^i_k)$.  \n3. The objective of Eq.14 is to *minimize* the distance between image tokens and their corresponding prompt indicators, ensuring that both $\\sum{\\mathcal{S}(\\boldsymbol{h}_m,\\boldsymbol{\\kappa}^i_m)}$ and $\\sum{\\mathcal{S}(\\boldsymbol{c}_j,\\boldsymbol{\\kappa}^c_j)}$ are as small as possible.  \n4. To enhance clarity, we have added an explanation of the optimization objective before Eq.14.\n### Q3: Fig.3\n1. Figure 3 in our paper visualizes the attention map in ViT, defined as $A=\\text{Softmax} \\left(\\frac{Q K^T}{\\sqrt{d_k}}\\right)$.  \n2. In (b) and (d), both the x-axis and y-axis are token indices, with the y-axis corresponding to queries and the x-axis to keys. The color variations indicate the attention weights.  \n3. In (a) and (c), the z-axis is the attention weights, while both the x-axis and y-axis correspond to token indices. The difference is that the x-axis represents queries, and the y-axis represents keys. Notably, for clarity, we did not visualize all queries but instead focused on CLS tokens and a subset of prompt and image tokens.  \n4. We have revised Fig.3, explicitly annotating the meaning of each axis. The updated figure is provided in Fig.1.2 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/1.2.png).\n### Q4: Notation\n1. In our original paper, $M$ represents the number of patches. It is **italic**, **uppercase**, and **not bold**, used to denote a *scalar*. In contrast, $\\mathrm{\\mathbf{M}}$ represents the final mask. It is **upright**, **uppercase**, and **bold**, used to denote a *tensor*.  \n2. To improve clarity, we have changed $M$, which originally represented the number of patches, to $N$, and $N$, which originally represented the number of network layers, to $L$.  \n3. The paper does not use subscripts $c$ and $h$. Instead, superscripts $c$ and $i$ are used to differentiate between CLS token-related and image token-related prompts and indicators. These are **lowercase**, **italic**, and **not bold**. Meanwhile, $\\boldsymbol{h}$ represents tokens in the network, which are *vectors* and are written in **lowercase**, **italic**, and **bold**.  \n4. To enhance readability, we have standardized the use of superscripts to indicate different attributes of tokens, prompts, and indicators (belong to CLS token or image token). Meanwhile, indices indicating layer numbers and patch indices are consistently placed in subscripts. For example, the notation $\\boldsymbol{h}^1_i$, which originally represented the first token in the $i$-th layer, has been revised to $\\boldsymbol{h}_{i,1}$.\n### Q5: Typo\n1. We have corrected \"SPT\" to \"VPT\" in Sec.5.5.  \n2. Additionally, we have thoroughly checked and revised other parts of the paper for grammatical issues and typos. For example, in Eq.9, we have corrected $\\boldsymbol{k} $ to $\\boldsymbol{\\kappa}$.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed approach contains several unresolved technical issues. The motivation is supported by previous existing works, yet the reason the proposed methods can assign different prompt to different image tokens is unclear. There is no theoretical analysis or strategy to ensure the load balance of different prompts. The eq.14 that optimizes keys of prompts looks wrong.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Sound.", "relation_to_broader_scientific_literature": "Two existing works [1,2] served as theoretical support to this paper.\n[1] Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.\n[2] Kim, Y., Li, Y., Moitra, A., Yin, R., and Panda, P. Do we really need a large number of visual prompts? Neural Networks, 177:106390, 2024.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "See other parts.", "other_comments_or_suggestions": "1.\tFigure 3 is difficult to interpret due to unclear axis labels. It should be immediately discernible which axis represents keys and which represents values. Revisions to these figures may needed.\n2.\tThe notation throughout the paper is inconsistent and poorly defined. For example, M is used to represent both the number of tokens and the final mask, which can cause confusion. The subscript c represents layers, while the subscript h represents index, which is counterintuitive.\n3.\tThe paper contains several grammatical errors and awkward phrasings that impede clarity.\n4.\tTypos in 5.5 (SPT).", "questions_for_authors": "I may raise my score if all my concerns are well addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZxSnpO9FYa", "reviewer": "Reviewer_zc31", "summary": "This paper proposes Token Coordinated Prompt Attention (TCPA) to enhance visual prompting for Vision Transformers. By disentangling and adaptively assigning prompts to different CLS and image tokens based on their distinct roles, this method effectively mitigates the limitations of conventional visual prompting and improves feature diversity and discriminability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our **motivation**, **effectiveness** and **comprehensive experiments**.\n### Q1: Theorem\n1. Theorem 4.1 and Theorem 4.2 mentioned in our paper are established theories from existing work, which we have *appropriately cited*.  \n2. In their original paper, these theorems were used to analyze the rank of the attention matrix in vision models. In our paper, we reference these theorems alongside Fig.3 to illustrate that existing prompt learning methods tend to focus on overlapping information, whereas our TCPA assigns different prompts to different tokens, *enabling a more comprehensive extraction of discriminative features*.  \n3. We have now included the proof of Theorem 4.1 and Theorem 4.2 in the appendix for completeness.\n### Q2: Hyperparameter\nThank you for your valuable suggestions. We have included hyperparameter experiments for the weight parameters $ \\lambda_i $ and $ \\lambda_c $, as detailed in **Reviewer LMp2  Q2:Hyperparameter**.\n### Q3: Section 5.5.4\n1. In Section 5.5.4, we primarily present the t-SNE visualizations of the features extracted by both the baseline methods and our approach, further demonstrating that our method captures more comprehensive discriminative information.  \n2. We have revised and refined the last two sentences in this section to eliminate redundancy.\n### Q4: Notations in Section 3\n1. We have thoroughly checked the formulas in the paper. For example, we have standardized the use of *superscripts to indicate different attributes* of tokens, prompts, and indicators (i.e., whether they belong to the CLS token or image token), while *indices representing layer numbers and patch indices are consistently placed in subscripts*. For example, the notation $ \\boldsymbol{h}^1_i $, which originally represented the first token in the $ i $-th layer, has been revised to $ \\boldsymbol{h}_{i,1} $. Additionally, to improve clarity, we have changed $ M $, which originally represented the number of patches, to $ N $, and $ N $, which originally represented the number of network layers, to $ L $.  \n2. Furthermore, we have carefully reviewed other parts of the paper and corrected grammatical issues and typos. For example, in Eq.9, we have corrected $ \\boldsymbol{k} $ to $ \\boldsymbol{\\kappa} $.\n### Q5: Optimization Objective\n1. The objective of Eq.14 is to *minimize* the distance between image tokens and their corresponding prompt indicators, ensuring that both $\\sum{\\mathcal{S}(\\boldsymbol{h}_m, \\boldsymbol{\\kappa}^i_m)}$ and $\\sum{\\mathcal{S}(\\boldsymbol{c}_j, \\boldsymbol{\\kappa}^c_j)}$ are as small as possible.  \n2. To enhance clarity, we have added an explanation of the optimization objective before Eq.14.\n### Q6: Eq.11\n1. The binarized matrix $\\mathrm{\\mathbf{\\hat{A}}} \\in \\{0,1\\}^{M \\times N_i}$ has dimensions matching *the number of image tokens*, but in the actual model, **the CLS token needs to be considered**. In other words, the dimensions of $\\mathrm{\\mathbf{\\hat{A}}}$ need to match *the number of image tokens plus one*. Eq.11 is designed to achieve this dimensional alignment.\n2. To help readers better understand, we have added an explanatory note before Eq.11 in the paper.\n### Q7: \"Row 9, Column 8\" in Figure 1\n1. The first two are objects, and a background token is selected as a reference for comparison to make the experiment more comprehensive. \n2. Existing methods focus on the same regions regardless of whether the tokens correspond to objects or the background, whereas in our approach, different object tokens attend to different object regions, and the background token focuses more on the background. \n3. This is because our TCPA matches different prompts to tokens with different semantics for attention interactions, enabling more comprehensive extraction of discriminative information from the image.\n### Q8: Notation $ p_{j+1}^{i,d} $ in Section 3.2\n1. This is a typo on our part. The output $p_{j+1}^{i,d}$ for each layer should be discarded and not used for the next layer's output.\n### Q9: Eq.9\n1. Yes, $k_{k}^{i}$ and $\\kappa_{k}^{i}$ in Eq.9 represent the same notation.  \n2. We have standardized it to $\\kappa_{k}^{i}$.\n### Q10: Notation $\\hat{A}_{m,k}$\n1. Yes, $\\hat{A}_{m,k}$ represents an element of the matrix $\\tilde{A}$.\n2. We have standardized it to $\\hat{A}$ and no longer use $\\tilde{A}$ to represent it.\n### Q11: Typo\n1. We have revised $\\mathrm{\\mathbf{M}}^i$ to $\\mathrm{\\mathbf{M}}^c$ in the sentence below Eq.11 in Section 3.3.  \n2. We have also carefully checked and corrected other sections of the paper for grammatical issues and typos, such as changing \"SPT\" to \"VPT\" in Section 5.5.", "claims_and_evidence": "Yes. Figure 1 supports the claim that existing visual prompting methods usually learn and leverage the same prompt for all tokens without considering the different functionalities of CLS and image tokens, as well as the varying discriminative information conveyed by different image tokens, leading to different tokens focusing on similar regions and extracting biased discriminative information.", "methods_and_evaluation_criteria": "Yes. The proposed method selects different prompts for different tokens and performs attention-based interactions, thereby improving the representation ability of ViT.", "theoretical_claims": "In this paper, Theorem 4.1 and Theorem 4.2 do not involve specific proofs.", "experimental_designs_or_analyses": "Yes. In Section 5.5.3., there is a lack of analysis of the two weight parameters \\lambda_i and \\lambda_c in Equation (14). Beside, the last part of Section 5.5.4. contains redundant experimental analysis.", "relation_to_broader_scientific_literature": "Prior methods use the same prompts for all tokens without considering the distinct roles of CLS and image tokens, as well as the differences in discriminative information extracted by various image tokens. This results in the features extracted by different tokens being neither distinguishable nor comprehensive, which limits the model's performance. This paper proposes TCPA to select different prompts for different tokens and performs attention-based interactions, thereby encouraging the model to extract more diverse and comprehensive discriminative information.", "essential_references_not_discussed": "No. The related work is enough to understand the research background of this paper.", "other_strengths_and_weaknesses": "Strengths:\nThe proposed TCPA addresses a key limitation of existing visual prompting methods, i.e., uniform prompt interaction, improving the diversity and representational capacity of the extracted features.\n\nWeaknesses:\n(1)The method description lacks sufficient clarity in Section 3.\n(2)The use of mathematical notation is somewhat inconsistent, which hinders comprehension.\n(3)The optimization objective lacks a detailed explanation, and the weight parameters are not supported by a thorough parameter analysis experiment.", "other_comments_or_suggestions": "(1)Please revise Section 3 of the paper to ensure clarity in the methodology and consistency in notation.\n(2)Please revise Section 5.5.4. of the paper to remove duplicate experimental analyses.\n(3)It is recommended to provide a more detailed explanation of Equation (11) and Equation (14) and include a parameter analysis of the weight parameters in the experimental section.", "questions_for_authors": "(1)What is the purpose of the attention region corresponding to \"Row 9, Column 8\" in Figure 1?\n(2)For the sentence “Through the equation above, we obtain the output for the image tokens (p_{j+1}^{i,d}, h_{1}^{j+1}, · · · , h_{M}^{j+1}), which, together with the previously obtained output of the CLS token c_{j+1}, serves as the input for the next MSA block B_{j+1}.” in Section 3.2, why is p_{j+1}^{i,d} not discarded.\n(3)Are k_{k}^{i} and \\kappa_{k}^{i} in Equation (9) representing the same notation?\n(4)Does \\hat{A}_{m,k} represent an element of the binarized matrix  \\tilde{A}?\n(5)Please explain Equation (11) and Equation (14) in detail, and add a parameter analysis of the weight parameters in the experimental section.\n(6)The sentence below Equation (11) in Section 3.3, “…, we obtain the mask M^{i} corresponding to the CLS token,” seems to contain a typo. M^{i} should be written as M^{c}.\n(7)There are repeated experimental analyses in Section 5.5.4.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JIWjv7sU8d", "reviewer": "Reviewer_LMp2", "summary": "The paper introduces Token Coordinated Prompt Attention (TCPA), a novel module for visual prompting in Vision Transformers (ViTs). TCPA assigns specific prompts to CLS and image tokens, enhancing their discriminative abilities through targeted attention interactions. It uses a matching function to dynamically allocate prompts to image tokens, improving feature diversity and representation. Experiments show TCPA outperforms state-of-the-art methods, validating its effectiveness in extracting comprehensive and discriminative features.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our **novelty**, **effectiveness** and **comprehensive experiments**.\n\n(The images mentioned below are available at the anonymous link: https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E.)\n### Q1: Cosine Distance-based Matching Function\n1. We conduct a visualization experiment on the image prompts selected by different image tokens. As shown in Fig.3.1 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/3.1.png), tokens corresponding to different parts of an object select different prompts. This indicates that our matching mechanism can recognize the different semantic information contained in tokens to some extent and assign the corresponding prompts accordingly. \n2. During the method design process, we explored various alternative matching strategies, including Euclidean distance, Kullback-Leibler (KL) divergence, and weight prediction through a learnable MLP. As shown in the table below, among these matching strategies, cosine distance achieved the best performance.  \n3. In future research, we will further optimize the prompt matching mechanism by incorporating neighborhood token information to achieve more accurate prompt assignment.\n\n||CUB|Dog|GTSRB|\n|-|-|-|-|\n|Euclidean Distance|89.3|91.4|93.4|\n|KL Divergence|89.2|91.3|93.6|\n|MLP|89.0|91.2|93.7\n|Cosine Distance|**89.5**|**91.5**|**94.1**|\n### Q2: Hyperparameter\n1. Thank you for your valuable suggestions. We conduct an ablation study on the weight hyperparameters $\\lambda_i$ and $\\lambda_c$ on the CUB dataset. As shown in Fig.3.2 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/3.2.png), the best performance is achieved when $\\lambda_i=0.03$ and $\\lambda_c=0.02 $.  \n2. When $ \\lambda_i $ and $ \\lambda_c $ are too large, they *interfere with the learning of prompts and the classifier*, leading to performance degradation.  \n3. When $ \\lambda_i $ and $ \\lambda_c $ are too small, *the indicators corresponding to the prompts cannot be effectively learned*, making it difficult to accurately match prompts to different tokens, which also results in performance degradation.\n### Q3: Base-to-novel Task\n1. First, *for a fair comparison* with existing visual prompt learning methods, we conducted experiments on the HTA and VTAB benchmarks to validate the effectiveness of our approach.  \n2. Existing visual prompt learning methods typically use a ViT backbone. When adapting a pretrained ViT model to downstream tasks, a task-specific classifier must be learned *based on the number of categories in the target task*. Consequently, due to the constraints of the classifier, ViT models cannot conveniently train on base classes and generalize to novel classes as CLIP models do.  \n3. Additionally, we adapt several visual prompt learning methods, including VP, VPT, and VFPT, and our TCPA, to the CLIP model to evaluate their performance on the base-to-novel task. As shown in the table below, using the VFPT method as an example, our TCPA improves performance by **0.5%–0.9% on base classes** and by **0.5%–1.3% on novel classes**. Similarly, our TCPA also achieves consistent performance improvements on the VP and VPT methods. This improvement stems from our approach’s ability to assign different prompts to different image tokens, enabling the model to capture more comprehensive and fine-grained discriminative information for each category, thereby enhancing generalization to novel classes.  \n4. In future research, we plan to incorporate part-level modeling of learned categories to further improve the applicability and generalizability of our approach.\n\n|Methods||Caltech101|OxfordPets|Stanford_Cars|Flowers102|\n|-|-|-|-|-|-|\n|VP|Base / Novel|97.0 / 93.5|92.3 / 94.1|68.4 / 72.8|89.5 / 70.2|\n|**+TCPA**|Base / Novel|97.4 / 94.1|93.1 / 94.4|70.7 / 73.6|95.5 / 71.2|\n|VPT|Base / Novel|97.3 / 93.9|95.3 / 94.3|72.4 / 74.1|96.2 / 71.1|\n|**+TCPA**|Base / Novel|97.8 / 94.6|95.7 / **94.9**|73.7 / 74.7|96.5 / 72.3|\n|VFPT|Base / Novel|97.6 / 94.4|95.9 / 94.2|73.4 / 74.6|96.6 / 71.3|\n|**+TCPA**|Base / Novel|**98.1** / **94.9**|**96.3** / 94.8|**74.2** / **75.7**|**97.5** / **72.6**|", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "For the Token Coordinated Prompt Attention (TCPA) module, the authors propose a novel approach to disentangle prompts for CLS and image tokens, enhancing feature diversity and discriminability. However, the prompt assignment process relies on a cosine distance-based matching function, which can capture the complex relationships between tokens and prompts.", "theoretical_claims": "Theoretical correct.", "experimental_designs_or_analyses": "The paper demonstrates the effectiveness of TCPA through comprehensive evaluations on HTA and VTAB benchmarks, consistently outperforming state-of-the-art methods like VPT and DAMVP across diverse tasks. Ablation studies clearly highlight the contributions of R-TCPA (role-level) and T-TCPA (token-level) components, with an analysis of prompt pool size. Visualizations, including t-SNE and attention maps, show that TCPA extracts more discriminative and diverse features compared to baselines. Additionally, theoretical analysis based on the low-rank properties of self-attention (Theorems 4.1 and 4.2) provides a solid foundation for TCPA's design.", "relation_to_broader_scientific_literature": "This paper advances the field of visual prompting by introducing the Token Coordinated Prompt Attention (TCPA) module, which enhances feature diversity and discriminability in Vision Transformers through role-specific and dynamically assigned prompts, addressing a key limitation in existing methods.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Incomplete Hyperparameter Sensitivity: Only prompt pool size is explored; other hyperparameters (e.g., weighting parameters in Sec. 3.4.) are not thoroughly analyzed.", "other_comments_or_suggestions": "None", "questions_for_authors": "How would TCPA perform if extended to the base-to-novel task [1,2] in prompt learning? It is recommended that the authors consider this direction for further research.\n\n[1] Khattak M U, et al. Self-regulating prompts: Foundational model adaptation without forgetting[C]//ICCV. 2023: 15190-15200.\n[2] Wu G, et al. Cascade prompt learning for vision-language model adaptation[C]//ECCV 2024: 304-321.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3xtlScko1", "reviewer": "Reviewer_2ZNj", "summary": "This paper proposes a Token Coordinated Prompt Attention (TCPA) module to enhance the effectiveness of visual prompting in Vision Transformers (ViT). Existing methods use shared prompts for all tokens, overlooking the distinct roles of CLS and image tokens, leading to limited representational capacity. TCPA addresses this by assigning CLS and image-specific prompts for targeted attention interactions, improving their discriminative abilities. A matching function further assigns coordinated prompts to individual image tokens to enhance feature diversity and representation. Experiments show that TCPA significantly improves feature diversity and performance.\n\n## update after rebuttal\nI have reviewed the authors' rebuttal as well as the comments from my fellow reviewers. I remain inclined to maintain my positive assessment and will keep my current rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our **novelty**, **effectiveness** and **comprehensive experiments**.\n\n(The images mentioned below are available at the anonymous link: https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E.)\n### Q1: More Visualizations\n1. Thank you for your valuable feedback. We have included additional 2D and 3D attention map visualizations with more samples in the appendix.\n2. As shown in Fig.4.1 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/4.1.png), the same trend is observed in the additional visualizations. In existing visual prompt methods, *the attention regions of prompts are highly similar*, leading to nearly identical features being extracted from the CLS and image tokens. In contrast, our proposed TCPA module *enhances more diverse attention across prompts, CLS tokens, and image tokens*. \n3. This is because our method selects different prompts for different tokens and performs attention-based interactions, encouraging the model to extract more diverse and comprehensive discriminative information.\n### Q2: Hyperparameter\n1. As shown in Fig.4.2 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/4.2.png), we have added hyperparameter experiments *on the Dog and GTSRB datasets*, which exhibit the same trend observed on the CUB dataset. \n2. When the prompt pool is too small, prompt diversity is limited, causing high overlap in selected prompts and making the extracted features indistinguishable. On the other hand, an excessively large prompt pool increases learnable parameters, which can lead to overfitting and decreased performance. Optimal performance is achieved with a moderate pool size.\n3. Additionally, we have included hyperparameter experiments for the weight parameters $\\lambda_i\n$ and $\\lambda_c$. As shown in Fig.3.2 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/3.2.png), the best performance is achieved when $\\lambda_i=0.03$ and $\\lambda_c=0.02 $. When $\\lambda_i$ and $\\lambda_c$ are too large, they *interfere with the learning of prompts and the classifier*, leading to performance degradation. When $\\lambda_i$ and $\\lambda_c$ are too small, *the indicators corresponding to the prompts cannot be effectively learned*, making it difficult to accurately match prompts to different tokens, which also results in performance degradation.\n### Q3: Pseudocode of the Method\n1. Thank you for your valuable suggestions. \n2. We have added the pseudocode of the method in the appendix to clearly illustrate the proposed approach.\n### Q4: Applicability of the Method\n1. Existing visual prompt learning methods can be categorized into two types based on the prompt placement: image-based and token-based. *Our approach is compatible with both*.  \n2. For token-based visual prompt learning methods, our TCPA can enhance the attention interaction process, thereby improving performance.  \n3. For image-based methods, token prompts can be incorporated alongside the original approach, integrating our TCPA to provide continuous prompting during feature extraction.", "claims_and_evidence": "The claims made in the submission are well-supported by clear and convincing evidence. The paper thoroughly validates the effectiveness of the proposed Token Coordinated Prompt Attention (TCPA) module through extensive experiments across multiple benchmarks. The authors provide detailed comparisons with state-of-the-art methods, demonstrating significant improvements in both feature diversity and overall performance. Additionally, ablation studies are conducted to isolate the contributions of CLS-specific and image-specific prompts, highlighting the effectiveness of each component.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria (e.g., benchmark datasets) are appropriate and well-suited to the problem at hand. The chosen benchmark datasets are representative, providing valuable insights into the model's performance across various scenarios.", "theoretical_claims": "The correctness of the theoretical claims in the paper is not in question. The paper relies on two established theories from existing literature to demonstrate that the self-attention matrix in current prompt learning methods is low-rank. Furthermore, through experimental analysis, it is validated that the proposed method enhances the diversity of the self-attention matrix, encouraging different tokens to focus on diverse discriminative information.", "experimental_designs_or_analyses": "The experimental design and analysis in the paper are both reasonable and effective. The authors have selected appropriate comparative experiments to validate the proposed method, and the benchmarks and evaluation metrics used in the experiments are well-targeted, allowing for a comprehensive assessment of the model's performance. Additionally, ablation studies are included to verify the performance of each module. The paper also presents several visualization experiments, further supporting the theoretical claims made. These design choices significantly contribute to the reliability and validity of the research.", "relation_to_broader_scientific_literature": "This paper introduces a plug-and-play enhancement module, TCPA, designed for existing visual prompt learning methods (such as VPT, VP, VFPT, etc.). The module works by modifying the interaction between the CLS, image tokens, and prompts in the attention mechanism, encouraging different tokens to interact with only specific prompts. This, in turn, helps the model extract more comprehensive discriminative information. \n\nTheoretically, the paper leverages two existing theories to demonstrate that the attention matrix in current prompt learning methods is low-rank. This is further validated through experiments, which also show that the proposed method enhances the diversity of the attention matrix, encouraging different tokens to focus on a broader range of discriminative information.", "essential_references_not_discussed": "No essential related works crucial for understanding the contributions of the paper have been omitted. The paper provides a comprehensive overview of related work in visual prompting, covering the core literature relevant to the proposed method. The authors have appropriately applied the relevant theories and provided a thorough introduction to them. These citations and discussions offer strong background support for understanding the key contributions of the paper.", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method is novel. The paper introduces Token Coordinated Prompt Attention for the first time, which changes the way tokens and prompts interact in prompt learning methods. By selecting specific prompts for each token to interact with, this approach helps the model extract rich and comprehensive discriminative information.\n2. The paper is well-structured and coherent. The logical flow is smooth, and the writing is clear. The accompanying figures effectively illustrate and validate the points made in the paper, making the arguments more accessible and convincing.\n3. The paper provides comprehensive comparative experiments. The authors conduct experiments across multiple benchmark datasets and integrate the proposed module into several existing prompt learning methods, achieving consistent performance improvements. This demonstrates the effectiveness and generalizability of the proposed approach across different scenarios.\n4. The ablation studies are well-designed. The authors include a wide range of ablation and visualization experiments, which help readers understand the role of each module and provide a clear visualization of how the proposed TCPA influences the attention maps and feature extraction process of the model.\n\nWeaknesses:\n1. The authors only show examples of 2D and 3D attention maps for a single sample. Providing additional examples across a wider range of samples would further strengthen the argument and offer additional validation of the method's effectiveness.\n2. The authors should conduct hyperparameter experiments on more datasets to thoroughly analyze the impact of hyperparameters on the model's performance. This would offer a more comprehensive understanding of how different settings influence the results.", "other_comments_or_suggestions": "It is recommended that the authors include pseudocode of the method to help readers better understand the process and flow of the proposed approach. This would make it easier for others to reproduce or build upon the work.", "questions_for_authors": "Could the authors clarify whether the proposed module is applicable to all visual prompting methods, or if it is specifically suited for certain types of methods, such as token-based prompting approaches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper received four positive ratings, with all reviewers generally inclined to accept it. The paper introduces a Token Coordinated Prompt Attention (TCPA) module, which is concerned with the unique role of different tokens in conveying discriminative information and assigns prompts according to the different roles of different CLS and image tokens. TCPA enriches discriminative information of tokens and improves the diversity and recognizability of visual prompt features. The paper is well-organized, providing comprehensive and convincing experiments and analyses to demonstrate its effectiveness. The design of the TCPA is supported by solid theoretical foundations and experimental evidence. However, some minor errors should be corrected, such as unclear labels on the axes of the figure, inconsistent symbols, and typos. The authors have addressed the concerns raised, resolving most of the doubts. Overall, it is a good work, and the Area Chair (AC) recommends accepting the paper.
Decision: Accept (poster)