Release the Powers of Prompt Tuning: Cross-Modality Prompt Transfer
Authors: Ningyuan Zhang
Jie Lu
Keqiuyin Li
Zhen Fang
Guangquan Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Cross-Modality
Prompt Transfer
Primary area: foundation or frontier models, including LLMs
TL;DR: Explore cross-modality prompt transfer to boost the performance of prompt tuning on data-scarce tasks

Abstract: Prompt Tuning adapts frozen models to new tasks by prepending a few learnable embeddings to the input.
However, it struggles with tasks that suffer from data scarcity.
To address this, we explore Cross-Modality Prompt Transfer, leveraging prompts pretrained on a data-rich modality to improve performance on data-scarce tasks in another modality.
As a pioneering study, we first verify the feasibility of cross-modality prompt transfer by directly applying frozen source prompts (trained on the source modality) to the target modality task.
To empirically study cross-modality prompt transferability, we train a linear layer to adapt source prompts to the target modality, thereby boosting performance and providing ground-truth transfer results.
Regarding estimating prompt transferability, existing methods show ineffectiveness in cross-modality scenarios where the gap between source and target tasks is larger.
We address this by decomposing the gap into the modality gap and the task gap, which we measure separately to autonomously select the best source prompt for a target task.
Additionally, we propose Attention Transfer to further reduce the gaps by injecting target knowledge into the prompt and reorganizing a top-transferable source prompt using an attention block.
We conduct extensive experiments involving prompt transfer from 13 source language tasks to 19 target vision tasks under three settings.
Our findings demonstrate that:
(i) cross-modality prompt transfer is feasible, supported by in-depth analysis;
(ii) measuring both the modality and task gaps is crucial for accurate prompt transferability estimation, a factor overlooked by previous studies;
(iii) cross-modality prompt transfer can significantly release the powers of prompt tuning on data-scarce tasks, as evidenced by comparisons with a newly released prompt-based benchmark.

Published as a conference paper at ICLR 2025
RELEASE THE POWERS OF PROMPT TUNING: CROSS- MODALITY PROMPT TRANSFER
Ningyuan Zhang, Jie Lu," Keqiuyin Li, Zhen Fang, Guangquan Zhang Australian Artificial Intelligence Institute, University of Technology Sydney ni Ingguaan.. zhangestudent. uts edu.. (jie. lu, keqiuyin.1,, zhen. fang, guangquan. sanggguun.hhhnggge (etti.juuuuuuuu edu.
ABSTRACT
Prompt Tuning adapts frozen models to new tasks by prepending few learnable embeddings to the input. However, it struggles with tasks that suffer from data searcity. To address this, we explore Cross-Modalin Prompt Transfer, leveraging prompts pretrained on data-rich modality to improve performance on data-scarce tasks another modality. As pioneering study, we first verify the feasibility cross-modality prompt transfer by directly applying frozen source prompts (trained on the source modality) to the target modality task. To empirically study cross- modality prompt transferability, to the target modality, thereby boosting performance and providing ground-truit transfer results. Regarding estimating prompt transferability, existing methods show ineffectiveness in cross-modality scenarios where the gap between source and target tasks is larger. We address this by decomposing the gap into the modality gap and the task gap, which we measure separately to autonomously select the best source prompt for a target task. Additionally, we propose Attention Transfer to further reduce the gaps by injecting target knowledge into the prompt and reorganizing a top-transferable source prompt using an attention block. We conduct extensive experiments involving prompt transfer from 13 source language tasks to 19 target vision tasks under three settings. Our findings demonstrate that: (i) cross-modality prompt transfer is feasible, supported by in-depth analysis; (ii) measuring both the modality and task gaps is crucial for accurate prompt transferability estimation, factor overlooked by previous studies; (iii) cross-modality prompt transfer significantly release the powers prompt tuning on data-scarce tasks, as evidenced by comparisons with newly released prompt-based benchmark.
train linear layer to adapt source prompts
INTRODUCTION
As pretrained Transformers Vaswani et al. 2017 become larger, the demand for transferring them to new tasks in parameter-effhiient way grows heavier (Peng et al. 2024). Prompt Tuning (Lester jt al. 2021) emerges as a leading Parameter-Efficient Transfer Learning (PETL) method, updating only the learnable vectors prepended to model inputs. It makes the least modifications the model's architecture compared to other PETL approaches (Guo et al., (2022), making it particularly compatible with complex or blackbox Transformer models, thereby warranting increasing popularity. However, some modalities struggle to benefit from prompt tuning. Data- scarce modalities, for example, lack sufficient training samples fully optimize the prompt (Su al.) 2022).
Our research seeks to tackle the challenges by exploring Cross- Prompt Transfer, aiming release the powers of prompt tuning on data-scarce tasks by transferring prompts pretrained on Natural Language Processing (NLP) tasks, which are well-suited for prompt tuning due to their data-rich nature (Lu et al.) 2022). Such exploration can greatly benefit data-scarce modalities by leveraging the rich data resources and well-established frameworks of the text modality through the transfer of text-pretr prompts, thereby unlocking the power of pretrained models and mitigating the challenges these modalities face.
"Corresponding Author

---

Published as a conference paper at ICLR 2025
In the literature, prompt transfer has been proven effective between NLP tasks (Vu et al. 2022 Su
ct al. 2022). We hypothesize that such practice can be safely extended to cross-modaile scenarios and conduct extensive studies to verify our hypothesis. We start with proving the feasibility of cross-modality prompt transfer and empirically studying the prompt transferability in cross-modality scenarios by designing two transfer settings: Frozen Prompt Transfer and Projection Transfer. Frozen prompt transfer directly applies text text-pretrained prompts (source prompts) to non-text tasks without modification, testing whether any transferability exists. For projection transfer, it enhances the performance of the frozen source prompts and obtains the ground-trute truth transfer results by training a linear layer to adapt to the target tasks that linguistic knowledge stored in the frozen source prompts.
With the ground bround-truth truth transfer results. we further study what determines the prompt transferability in cross-modality scenarios by analyzing the gap between source and target tasks. Observing that existing methods for in-modality scenarios encounter limitations, we hypothesize that the gap between the source and target task is more complex in cross- cross-modality scenarios. response, we identify two gaps source prompt must overcome to be employed on a different modality task: (i) the modality gap formed by the differences in data distributions and models between the source and target modalities, and (ii) the task gap formed by the differing nature of the tasks themselves. By measuring these gaps, we propose a novel prompt transferability estimation approach that autonomously select the best source prompt more accurately compared to current methods designed for in-modality scenarios.
Based on the analysis of the modality gap and task gap, we further explore whether the performance of prompt tuning can be further boosted by bridging these gaps. In response, Attention Transfer is proposed. It reduces the gaps by injecting target knowledge into a highly transferable source prompt and by an attention block that adapts the source prompt more effectively. As a result, attention transfer can boost prompt tuning to a level comparable to or even better than the newest prompt tuning benchmark. demonstrating the potential of cross- moossssyy prompt transfer. In summary, our contributions can be summarized as:
We explore cross-modalls prompt transfer as an effective approach for boosting prompt tun- ing and verify the feasibility through extensive experiments and in-depth analysis, addressing a critical research gap and opening new possibilities for data-scarce modalities.
We introduce novel method for estimating prompt transferability by quantifying the modality and task gaps, which the existing in-modality methods have overlooked, as the gaps are enlarged in the cross-modality scenario. As result, our method offers a more accurate solution compared to existing in-modality methods.
We further demonstrate the powers of cross- modsslttyy prompt transfer through Attention Transfer, which eases the modality gap and task gap by injecting target knowledge into the prompt and utilizing a top-transferable source prompt more effectively. As a result, attention transfer enables prompt tuning to perform comparably or even better than the best prompt-based benchmark.
RELATED WORK
Caaaneter-Eflicienn Transfer Learning. As model size increases, finetuning becomes infeasible for adapting pretrained models to new tasks. Therefore, PETL approaches that steer pretrained model by tuning only small amount of weights are favored, they help avoid storing different model instances for individually varied tasks (Ding et al. 2023). Most PETL methods are built upon three baselines: (i) Adapter (Houlsby et al. 2019) that inserts learnable projectors between layers, (ii) Low-Rank Adaptation (LoRA) (Hu et al. 202T) that trains rank decomposition matrices as the update matrices for model weights, and (ii) Prompt Tuning (Lester al.) (021) that concatenates learnable embeddings with input embeddings. Among these basclines, prompt tuning acquired many of popularity and had been extended to various pretrained models (Wang et al. (024a) due to its ability to steer architecturally complex or black-box models, as it makes the least modifications to the model architecture. As an influential follow-up study, JJa et al. (2022) extended prompt tuning to the Computer Vision (CV) domain and proposed Visual Prompt Tuning (VPT), integrating prompt tuning with Vision Transformers (Dosovitskiy et al. (020). Following VPT. Wang et al. (2024b) explored the influence of prompt initialization to VPT: after empirical verification that the mutual information between prompt and image patch embeddings tends to increase as prompt tuning proceeds, they proposed Self-Prompt Tuning (SPT) that initializes the prompts with sampled image
2

---

Published as a conference paper at ICLR 2025
Sourcce Fronce
Frioes Fronge Transfer
Transfr
RoBERO
Orapul Phodikee It,
VIT
Saaroo Proegt P.
Tea Eniieddegs
Fronce Searce Proree Ps
Poech Eeboodings
Proected Nemmet/pp)))
Pacch Imbheddrage
RuBERTA Teatelllr
Irage Trchedder
Pharge Prajaer
Irage bebaader
Daa
Figure 1: (a) Source Prompt Tuning: Perform prompt tuning on NI.P tasks with RoBERTa to get the source prompts. (b) Frozen Prompt Transfer: Source prompts used as-is target vision tasks to verify the feasibility of cross-modality prompt transfer. (c) Projection Transfer: Source prompts are projected and prepended the patch embeddings, to adapt the source knowledge.
patch embeddings. SPT demonstrated that VPT can be boosted significantly if properly initialized and achieved a new benchmark for prompt-based PETL methods. Our investigated cross-modality prompt transfer can be regarded as initializing the prompt with adapted NLP- NLP-pretrained prompt. and SPT both focus on prompt initialization but are from different perspectives. Therefore. to demonstrate the effectiveness of cross-modality prompt transfer, SPT is chosen for comparison.
Prompt Transfer. Although parameter-efficient, prompt tuning still falls below full-finetuning. Moreover, training prompt is often slower than finetuning and prompts tend to be less stable during optimization (Li & Liang (021). Therefore, researchers resort to prompt transfer to boost the performance and promote stable optimization for prompt tuning. Prompt transfer first trains source prompt on source task, before it initializes the target prompt with the trained source prompt for a target task. Vu et al. (2022) first explored prompt transfer between different NLP tasks and used the similarity between prompts to measure the prompt transferability the different tasks share. In addition, (Su et al. (2022) explored prompt transfer between different language models and used projection module to mitigate the discrepancy resulting from different language models. They also used the overlapping rate of activated neurons in the language model to measure the prompt transferability, which is more effective than the similarity measurement. Vu et al. (2022) and (Su et al.) (2022) both verified that prompt tuning can benefit from pretraining the prompts on an intermediate task. However, whether the conclusion still holds if the intermediate task comes from different modality remains underexplored. In response, we explore the prompt transfer between not only different models but also different modalities, filling the gaps left by previous studies.
METHODS
3.1
CROSS-MODALIIS PROMPT TRANSFER
Visual Prompt Tuning. Before diving into cross cross- modality prompt transfer, the basic concept of VPT needs to be introduced. VPT extends the concept of prompt tuning (Lester et al., [02T] from
language vision and serves as representative PETL method for ViT (Dosovitskiy al. 2020).
VPT requires a target vision task T, = { Yt and pretrained ViT consisting of an image embedder
E. and backbone B,. The goal of VPT is to minimize the empirical error on the target vision task:
arg min C [H. (p E, (2:)), vel p.
(1)
Where represents the prompt embeddings, H, represents the classification head, and stand for function and vector concatenation, respectively. Learnalle modules are colored in orange.
Frozen Prompt Transfer. To verify the feasibility of cross-modality prompt transfer, frozen source prompts Pa are prepended to the patch embeddings and fed to the ViT (Figure [p). Linear probing if frozen source prompt can help
(Oord et al. 2018)) is performed on top of the [CLS] features VIT form improved feature clusters, compared to a randomly initialized prompt and vanilla linear probing where no prompts included. In frozen prompt transfer, the only trainable module is the
3

---

Published as a conference paper at ICLR 2025
output predictor Hy, leaving the learning objective in Equation to be reformulated into:
arg mrg min & [H Bp(Ps |1 E v1))
Hu
Projection Transfer. To adapt the linguistic knowledge stored in the source prompt to CV tasks, source prompts are kept frozen but passed to a Iearnable one-layer linear projector P before being prepended to the image embeddings (Figure (p) During the transfer, only the output predictor H, and the projector are trainable, reformulating objective Iinto:
arg min & [H, o B. || (o())))
Ha.P
In the literature, Merullo et al. (2022) verify that the semantic spaces of language and vision can be channeled simply through a linear projector. It, therefore, becomes interesting explore in cross-modality prompt transfer, whether the source and target prompt spaces can be connected via a linear layer, the use of which can also be regarded as helping source prompts overcome both the modality and task gaps: smaller gaps would generally yield better performance. Therefore, projection transfer is investigated study cross- cross- modallty prompt transferability.
3.2 PROMPT TRANSFERABILTY ESTIMATION
Prompt transferability estimation aims to select the best source prompt for given target task without going through repetitive prompt tuning processes with all the source prompts. Existing methods achieve this by calculating the cosine similarity (Vu al. 2022) or model activation similarity (Su et al. 2022) between the source and target prompts. However, these methods are designed for the in modality scenario, where the modality gap is negligible and the task gap introduces the main discrepancy between source and target tasks. Their effectiveness encounters limitations in the cross-modality scenario as the modality gap enlarged. Therefore, estimating the cross-modality prompt transferability, the key lies in measuring the modality gap and task gap.
Modality Gap. The modality gap is interpreted as the distribution difference between source and target data that comes from different data types. The difference between source and target models and modalities causes it. In practice, directly estimating the distribution gap between source and target data is non-realistic because: (i) it computationally inefficient as some tasks might contain tons of samples, and (ii) different source datasets have similar data embeddings because they are sampled from the token embeddings of the same language model, leading to insufficiently distinct distributional differences between all the source datasets and particular target dataset. The source prompts, on the other hand, can effectively exploit and condense information from the source data as stated by Lester et al. (202T). and Zhong et al. (2021). Therefore, the source prompt is treated as an abstraction of the source data which it will replace to participate in calculating the modality gap. As result, the modality gap Gas between a source NLP task and target CV task t is estimated the Maximum Mean Discrepancy (MMD) (Greetonn et al. 2012) between the source prompt and target image patch embeddings (more details can be found in the Appendix):
(M( () = MM (xa))
Task Gap. Existing prompt transferability estimation methods (Vu et al. 2022 Su et al. 2022
are designed for in-modality scenarios where the modality gap is negligible. Therefore, they can be regarded as estimating the task gap by measuring the similarity among prompts trained on different tasks, treating the trained prompts as task embeddings. However, in cross- cross-modality scenarios, directly measuring the similarity between source and target prompts could lead to poor estimation of the task gap, as the source and target prompts lie different semantic spaces. To this end, we propose a simple yet effective method that trains universal linear projector to cast the source prompts to the target prompt space, enabling existing methods to function under the cross- modality scenario.
Specifically, given target CV task, we first perform vanilla VPT to obtain the target prompt. Then, source prompts trained on different NLP tasks are shuffled with each other, after which they are sent to the universal projector P. which consists of single linear layer. P. is learned by minimizing the Euclidean distance between the sefffle-then-rrojeette source prompts and the target prompt (more details can be found in the Appendix). Note that, although they share the same architecture,
4

---

Published as a conference paper at ICLR 2025
the universal projector P. used here is functionally different from the linear projector P used in projection transfer. P is learned from scratch in every source-targe pair (works only on this pair) and its goal is to minimize the target error, but P., is learned from every target task (works on every source prompt) to minimize the vector distance.
After Pa is trained, the task gap Gr between an NLP task and the given CV task is measured by:
+ EEOD
(T( = -cos( (co,))=-o(((
(5)
Where c: represents the cosine similarity between two vectors, p and p are the i-th vector of the source and target prompt respectively, and is the number of vectors in the source and target prompt.
Prompt Transferability. With the modality gap Gm and task gap GT defined, the cross-modality prompt transferability between a source NLP task s and a target CV task t is then estimated by combining Gm and Gr:
T(s,t) (s,
(rr(s gr(stt)
(6)
3.3 ATTENTION TRANSFER
To further help the source prompts overcome both the modal- ity and task gaps, we propose a novel cross-modality prompt transfer scheme named attention transfer (Figure 2). It eases the modality gap by selecting a source prompt with a small MMD value to the target data, and by injecting target knowl- edge into the final prompt. For the task gap, it uses a Prompt Concentrator (C) to elicit the source knowledge from the source prompts in a more complex but effective way.
Osiput Predictor:
VT Rackbener 8,
c(pp)
Puxch Embeddings
Prompt Concentntor
Image Embooder E,
Frozen Source Prompt P.
Image Patchex
The prompt concentrator takes the frozen source prompt P. (length I) as input and outputs a concentrated and adapted source prompt of length l, (l, 1) by the attention mecha- nism. It has three key components: (i) the query Q E IR xd%. that decides the importance of each vector in the source prompt, (ii) the key projection matrix W' that projects the source prompt to the space of Q for query operation, and (iii) the value projection matrix W. that adapts the source prompt to the target task. Wa and W, are in Rdh xdh, where R is the hidden dimension of the language/vision model. The output is obtained by the conventional attention operation:
Figure 2: Attention Transfer.
C(p.) softmar( x (p+ Wa) (ps W.)
The prompt concentrator not only better arranges the source prompt, but also makes space for target knowledge. After the source prompt is concentrated, it is concatenated with a sequence learnable prompts (randomly initialized, termed the target prompt) form the final prompt for the target task.
These target prompts aim at learning pure target knowledge, with a length of 4, (4 = I (1-1,) (.).
In attention transfer, the trainable modules are the prompt concentrator, target prompt, and output predictor, with the following learning objectives:
arg min L [H, o B.(C(p.) p: (xx)). Ha.Cop
(8)
In the literature, Wang et al. (2023) verified that prompt tuning the query, key, and value in an attention block can help a pretrained model learn new attention patterns required by a target task. Our attention block has similar aims but differs in detail: it also adapts pretrained modules to target tasks through learning new attention patterns. But only involves the query prompt, as its key and value matrices are not frozen, enabling attention patterns to be grasped without key and value prompts.
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
To study cross-modality prompt transfer thoroughly, a wide range of NLP tasks and CV tasks are involved. Source prompts are trained using a pretrained language model RoBERTa (Liu al., 2019) on each NLP task with a uniform length of 100. On target CV tasks, a pretrained VIT is used. The final prompt length is kept at 100 for all transfer scenarios and vanilla VPT.
ViT is a classic and representative Transformer-basee vision model, and VPT was initially exper- imented on VIT. Therefore, ViT selected as the target vision model. RoBERTa is an encoder language model that shares nearly the same architecture to ViT. Thus, it might share similar prompt space to VIT. Therefore. RoBERTa is chosen as the source language model. Note: (i) Only the base-model sizes are explored (e.g. RoBERTa-base and ViT-base which all have weight amount of 10-2224-n222
pretrained checkpoint is explored. This ViT checkpoint is pretrained in supervised manner on ImageNet-221 (Deng et al., 2209). with an image size of 224 x 224 and a patch size of 16 x 16. The exploration of different model sizes, different source language models, and diverse VIT pretraining schemes will be left to future work.
~ 1225M). (ii) For ViT, the vit-base-paa
Source NLP Tasks. In total, 13 investigated NLP tasks are selected as the source tasks to train the source prompt. These tasks can be categorized into four groups: (i) Sentiment Analysis (SA) tasks, including IMDB (Maas et al,, 201T), SST-2 (Socher et al. 2013), laptop (Pontiki et al. 2016). restaurant (Pontiki et al.. 2016). Movie Rationales (Movie, Zaidan et al. (2008)). and Eveet (Tweet, Barbieri et al. (2020)) (ii) Natural Language Inference (NLI) tasks, including MNLI
(Williams et al. 2017). QNLI (Wang et al. 2018), and SNLI (Bowman al. 2015). (iii) Ethical
Judgment (EJ) tasks, including deontology and justice (Hendrycks et al. 2020). (iv) Paraphrase
Identification (PI) tasks, including QQP (Sharma et al. 2019) and MRPC (Dolan & Brockett 2005).
Target Vision Tasks. The VTAB-IK ZZhai al. 2019) image classification tasks are chosen as the target tasks. VTAB-IK consists of 19 diverse image classification tasks, each with training set of 1000 images. These tasks can be divided into three main categories: (i) Natural tasks (including
CIFAR 100 (Krizhevsky et al.] 22099), 101tech101 (Li et al., 2004). DTD (Cimpoi et al. 2014).
Flowers 102 (Nilsback Zisserman 2006), Pets (Parkhi et al. 2012). ). SVHN (Netzer et al. 201
and SUN397 (Xiao et al. 20100) that contain natural images captured using standard cameras; (ii)
Specialized- tasks (including Patch Camelyon (Veeling al. 2018), EuroSAT (Helber et al. 2018).
Resisc45 (Cheng et al. 2017), and Retinopathy (Dugas al. 2015)) that contain images captured
via specialized equipment; and (iii) Structured tasks (including Clevr (Johnson et al. 2017). DMLab
(Zhai al. 2019). KITTI (Geiger al. 2012), dSprites (Matthey al. 2017). and SmallNORB LeCun et al. 2004) that require geometric comprehension like object counting.
Implementation Details. For vanilla linear probing and frozen prompt transfer, the original VPT repository released a set of linear probing hyperparameters that were carefully grid-searched on each CV task. The same hyperparameters are adopted for vanilla linear probing and frozen prompt transfer, as the nature of the frozen prompt transfer identical that of linear probing.
For vanilla VPT, we follow the procedure of Jia et al. (002) performing grid-search on learning rates (0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10) and weight decay values (0, 0.0001, 0.001. 0.01), with a batch size of 64, warm-up steps of 10, cosine learning rate scheduler, and an SGD optimizer with a momentum of 0.9. Note that following Jia et al. (2022). the learning rate is multiplied by (Batch Size/288 256) before training starts. We use the official 800-200 split released by Zhai et al. (2119) to perform the grid-search, training on 800 images and validating using the remaining 200.
For projection transfer, batch size of 64, a learning rate of 0.005, and weight decay of 0.001 are used. For the optimizer, Adam (Kingma & Bal 20144) is adopted.
For attention transfer, we first perform a grid-search to find the potentially best source prompt and its concentrated length 4. Note that the source prompt is searched only from the top- transferabeee ones and the hyperparameters of projection transfer are adopted during this stage. Then, another round of gearch using the potentially best source prompt and I, to find the potentially best learning rate from 0.5, 0. 0.05, 0.01, 0.005, 0.001, ..000.0 and weight decay value from
perform

---

Published as a conference paper at ICLR 2025
() Relative Performance of Fruzen Prompt Transfer w.nt Randomly Inittalized Prompt
(a.2) Relative Performance of F Frozen Prompe Transfer w.r.
Vanilla Linear Probing
imdo
5502
laptoo
800
restourant 3.89 1:0 146
3.00
1.01
1.0
5.00
140
1.00
0.0 3
1.4 3:00
1.03
1/0
movie j.z LoI 140
141
1.0
140
1.1
144
030
twoet XXI 2.0 1.6 3:1 1.00
135
140
10
1/2
0.5
0.5
K00
NON CO6
mnl--a 1.0
140
035
6
500
K.0
onb- 1. -MM
140
033
deontology
justice
impe-1a9
Xavier- -.00 LX AuD 1:00 Los 1.9 3:00
1.9
100
1.0
LOE
1.00
1
140
3.00 1.0 140
5:9
0.5
500
1
100
029
1.0
054
6:00
305
4.M
imdo 1.8t 5.9 1.40 3:00
1.00
141
1.4
1.00
140
33T
5:3
1:2
1.34
1.08
1.0
1
3:11
1,00
1.00
1:44
5002 (.11 Lan 1.0
140
1.11
3.22
1
111
laptoo- 3.99
3.11
restaurant
movie
mnli:
anli
3.95
1.00
3.56
3.69
3.0
135
snt 3:00 1:00 126
144
3.87
1:00
1.00 315
1.44
5.0
1.8
3:11
1:
029 1:77
deontnlogy 1.21 --0 100 3.30
1.0
140 1.8
1.0
140 3.0
1.0
100
1.0
justice 1.1 200 1.0
3:00
1.00 1.09 315 1.0 3.5
1.0
1.0 3:30
500
125
500
125
1.00
1.0
1.0
1.00
1:00
1.04
1.00
1
19
egp xas Son KAd 3:21 1.00
1:0
1,09 3.8
1:0 141 114 1.43
5.00
3.23
3:0 1.0 121 5:04
1.58
L33
124
mipe-u kaa 1.20
2.30 LOE 149 2.30
100
afterd noversLe
oxtont, L IPrpet
cleo moort
ciler
82
IR pat
dmiss
dlor count
adon foo
(b.1) Relative Performance of Projection Transfer w.rt
(h.2) Relative Performance Projection Transfor w.st
Randomly Initialized Prompt
Vanilla Visual Prompt Tuning
Figure 3: (a) Relative Performance of Frozen Prompt Transfer concerning (1) a random prompt, and (2) vanilla linear probing. (b) Relative Performance of Projection Transfer concerning (1) a random prompt, and (2) vanilla VPT. The vertical axis shows the NLP tasks on which the source prompts are trained, while the horizontal shows the CV tasks the source prompts are transferred to. xavier means the prompt is not trained on an NLP task but randomly initialized using Xavier initialization (Glorot & Bengio| 2010) and also kept frozen during transfer. The relative performance is calculated by (Performance B x 100% where B is the performance of transferring random prompt, conducting vanilla linear probing or VPT, depending on the situations stated in the figure.
{0, 0.0001, 0.001, 0.01). with batch size of 64, warm-up steps of 10, cosine learning rate scheduler, and an Adam optimizer. The complete set of hyperparameters of different scenarios on each CV task is listed in the Appendix.
4.2 GROSS-MODALITS PROMPT TRANSFER
In this section, frozen prompt transfer is first carried out to verify the feasibility of transferring prompts across modalities. The results are reported in Figure where the relative performance compared with two different baselines is reported: (i) frozen prompt transfer using a randomly initialized prompt, and (ii) vanilla linear probing (re-ran by us).
Later on, to adapt the linguistic knowledge stored in the source prompts to CV tasks and study the cross-modality prompt transferability, projection transfer is carried out, with the results reported in Figure 3p. Similarly, the performance is reported in two different formats: (i) the relative performance concerning projection transfer with randomly initialized prompt. (ii) the relative performance pertaining to vanilla VPT (with a fixed prompt length of 100, re-ran by us). The exact accuracy of linear probing, VPT, and projection transfer (only the accuracy with the best source prompt) is
7

---

Published as a conference paper at ICLR 2025
reported in T All other accuracy results are left in the Appendix. Based on Figure 3 and Table [ the following key conclusions are drawn:
(i) Source prompts pretrained text data can be safely transferred to tasks in different modality. This claim is supported by the fact that the trained prompts can help pretrained ViT achieve better linear probing performance compared to random prompt and vanilla linear probing (Figure Ba),, inferring that the feature clusters extracted by the ViT model will benefit from simply prepending the frozen source prompts the image embeddings. When compared with a random prompt, 150 out of 247 transfer pairs (61.5% achieve better performance. This number decreases when compared with vanilla linear probing, but still holds value of 78 (31.6%), demonstrating the potential for transferring NLP prompts across modalities.
(ii) The linguistic knowledge stored in the source prompt is helpful on a different modality task after adaptation. In Figure 3,. 86.2% of the transfer pairs (213 out of 247) can achieve better performance compared to conducting projection transfer using a random prompt. This demonstrates the importance of the linguistic knowledge stored in the source prompts, which is obtained through pretraining the source prompts on NLP tasks. In Figure D..2 64% of the transfer pairs (158 out of 247) are better than vanilla VPT, with the largest boost observed on SNLI-to- -is-tt--DDprrtts/lccc This demonstrates that the linguistic knowledge learned by the source prompts can be of great help on a different modality task after adaptation. Additionally, the improvements observed in Figure 3.2 are proven to be statistically significant: they are caused by transferring the source prompts, instead of error or random chances (the prove can be found in appendix).
(iii) More source data should yield better transfer performance, but this is not always the case. Normally, in transfer learning, more source data would bring better performance
the target task. This pattern also holds on projection transfer: performant source prompts (best first: SNLI, MNLI, QQP, and QNLI) tend have been pretrained on a vast amount of text data; 549K sentence pairs for SNLI: 392k sentence pairs for MNLI; 363k sentence pairs for QQP: and 104k sentence pairs for QNLI. The opposite also runs true: the worst source prompts (laptop. restaurant, and movie) are only pretrained on less than 3k sentences. This general pattern holds basically for every target CV task. However, the volume of pretraining data does not solely determine the transferability of the source prompts, as plenty of counterexamples can be found in Figure 3p (SNLI does not always achieve the best performance on the target CV task). To this end, we take a different perspective that measures both the modality and task gaps the cross modality prompt transferability indicator.
(iv) Prompt tuning performance on target tasks that already benefit a lot from prompt tuning is likely to be further boosted by cross-modality prompt transfer. This claim is drawn from the relation between
Pmoj Trans APF
V/L and P/V in Table V/L shows the relative performance gain of VPT over linear probing (V/L gain). P/V shows the relative performance gain of projection transfer over VPT (P/V gain). To visualize the relation more intuitively, we plot the values of V/L gain and P/V gain in Figure 4 which shows clear positive relation between V/L gain and P/V gain. Looking deeper into the values in Table tasks with V/L gain lower than 3% have average P/V gain of only 1.7% while those with a V/L gain greater than 3% come with an average P/V gain of 14.1%. Although counterexamples or outliers (such as dSprites/loc) can found, the overall trend still informs that, if task benefits lot from prompt tuning, then its prompt tuning performance is likely to be further boosted by cross mross-modality prompt transfer.
Figure 4: Scatter plot of V/L gain vs P/V gain. The values are log- scaled for better visualization.
4.3 PROMPT TRANSFERABILLT ESTIMATION
For a target CV task, we obtain the transferability scores for each source prompt by two steps: (i) Train the target prompt (with random initialization) the target task using vanilla VPT and (ii) Perform Equation to obtain the transferability score for each source prompt. Note: when training the universal projector for measuring the task gap, learning rate of 1c-4, weight decay of 0.1, batch size of 13 (all source prompts are loaded in the same batch), and the Adam optimizer are used. This set of hyperparameters is grid-searched on CIFAR100 and extended to every target task. The universal
8

---

Published as a conference paper at ICLR 2025
Table I: Accuracy of linear probing (Linear), visual prompt tuning (VPT), and projection transfer (Proj.). V/L = (VPT- - Linear) Linnar x 100% shows the relative gain of VPT over Linear. P/V = (Proj / /VPT> x 100% shows the relative gain of Proj. over VPT. Note: (i) The VPT re-ran by us uses a uniform prompt length of 100 on all tasks. (ii) For Proj., only the result obtained by the best
source prompt is reported.
C3TAR0
Thwee 102
Gombrad
077
560377
SVIN
TMN
AFTTI
22
539
V27
16.7
05.
95.
202
71.3
108
13.7
92.
740
30.5
12.3
27.6
228
1200
300
51.6
32.0
+4.86
-9.55
+47.05
Peg.
83.0
810
05.4
1823
890
32A
828
510
82.0
953
74.
436
17.0
23.6
28.3
12.5
414
38A
625
projector is trained for 100 epochs and only the last-epoch universal projector is used. More training details can be found in the Appendix.
After the transferability scores of all the source prompts are obtained for target task, the scores are ranked from highest to lowest. The ranking results are compared with the ranked projection transfer results on the target task with Kendall's coefficient (Kendall 1938) as the evaluation metric. Kendall's 's coefficient is often used to determine the strength and direction of the relationship between two ranked variables. It ranges from -1 1. Greater values indicate better transferability estimation results. The Kendall's coefficient for the investigated target tasks is listed in Table which involves the following methods:
Average cosine similarity (Avg Cos) that average-pools the source prompt vectors into one vector and calculates the cosine similarity between the average-pooled source and target prompts. This method is investigated by Vu et al. (2022) and Su et al. (2022).
Model stimulation similarity (ON, introduced by Su et al. (022 ) that feeds the source and target prompts to the language model separately and calculates the overlapping rate between the attention block's activation patterns of the source and target prompts. our case, since the source prompt will become part of ViT's input, we feed the source and target prompts to ViT and obtain the activation map of the last three attention layers of ViT calculate the overlapping rate, given that Su et al. (2022) had verified that using the last three layers would give the best result.
Vanta that ranks the source prompts according to the data amount they were pretrained on. This simple method is involved a support for a former claim that "more source data would yield better transfer performance", showing the high relevancy between the amount of pretraining data and cross- modos-ty prompt transferability. Despite the high relevance, Vaate cannot serve as reliable prompt transferability estimation method, as the ranking results are determined purely by the attributes of source prompts, leaving no target information taken into consideration. Therefore, Vánta will not be involved in the comparison.
GM Only and Gr Only use the modality gap GM or task gap Gr as the transferabilityy metric. We apply these two baselines to verify our hypothesis and show the importance of estimating the modality gap and task gap concurrently.
From the results in Table 2 it is clear that by measuring both the modality and task gaps at the same time the cross-motee mosdalyyyy prompt transferability can be well estimated. Looking deeper, we can observe that measuring only GMr can obtain a result much better than measuring only Gr and is very close to Gas & Gr. We hypothesize that in cross- modoliyyy scenarios, the modality gap would normally be more influential to the transfer performance than the task gap. Moreover, it's interesting to note that the only difference between Avg Cos and GT Only is the involvement of the universal projector, without which the cosine similarity cannot serve as solid approach for estimating cross-modality prompt transferability. This verifies that source and target prompts lie in different semantic spaces and direct similarity caleulation will not be meaningful. For the baseline method ON, it achieves relatively good results on most of the target tasks, even if the source and target prompts lie in different semantic spaces, demonstrating that it's a promising direction for diving deeper in the future.
9

---

Published as a conference paper at ICLR 2025
Table 2: Kendall's coefficient (scaled to -100 100) calculated each target task by different prompt transferability estimation methods. Combining the modality and task gaps yields the best transferability estimation result. The best results are bold while the best but equal results are underlined (Vdata that ranks the source prompts based on pretrained data volumes will not be included for comparison). The exact ranking results can be found in the Appendix.
Clbechill
SSrricatte
Clesidas
DMLLON
LTD
Meon
Ava Cos
1.28
33.33
12.82
19.23
0.00
6.41
11.58
0.00
10.25
30.77
32.05
51. 25
66.67
45.15
25.08
-0.53
-2.21
35.90
30.77
-10.66
6.41
61.54
1.80
62.82
17.95
11.19
23.06
-38.07
-7614
1.35
41.44
0.00
46
12.92
-17.5
58.91
14.17
73.08
87 18
$7.05
M4.7
64 10
70.51
11
90.91
76
71
19
61.10
$6.60
61.10
$2.15
89 18
80 54
55.19
Oy Only Only
15.64
92.51
88.34
78.21
64.10
78.11
80.77
14.36
87.18
65.36
89.14
$6.82
09.67
$7.15
56.41
$6.92
89.24
92.31
2983
62.32
76.79
64.10
70.51
51.89
87.18
$2.06
82.05
11.79
71.54
56.02
94.87
Table 3: Accuracy of Self-Prompt Tuning (SPT) and attention transfer (Atm.). Note: (i) the
accuracy of the baseline SPT is obtained from its original paper. (ii) The best accuracy is bolded.
Natual
Specialion
Canalyoa
SmalINDRRoai
SmallNORDole
Canechiul
CIRARIOR
Fooess 122
Rotinopphy
4Spntesion
Clevibaur
Rosbet
Clevndlic
SVHN
Mem
KITII
Men.
1711
SPT
91.2
78.9
71.2
99.4
90.7
52.3
56.4
81.44
52.5
82.6
94.9
74.6
83.65
48,0
6888
24.4
37.4
72.6
41.9
61.1
68.6
52.86
Ama.
90.1
$1.7
68.9
98.4
89.4
52.6
32.5
80.55
$2.1
81.
96.3
75.3
84.12
44.9
77.4
24.3
36.3
73.1
432
60.8
67.9
53.47
4.4
ATTENTION TRANSFER
To further demonstrate the effectiveness of cross-modality prompt transfer, attention transfer is compared with the newest prompt-based benchmark: Self-Prompt Tuning (SPT, Wang et al. (2024b)). SPT focuses prompt initialization for boosting prompt tuning: it initializes the prompts with sampled image patch embeddings, which can be regarded as injecting prior target data knowledge into the prompt. Similarly, attention transfer prompt: the final prompt of attention transfer contains knowledge not only from the source NLP task but also from the target CV task. Due this similarity and the timeliness, SPT chosen as the comparison baseline. The results are reported in Table
also be regarded injecting knowledge into the
From the results: attention transfer can achieve comparable average performance on Natural tasks and better average performance Specialized and Structured tasks. This serves as strong evidence for cros--modaiity prompt transfer being able to boost prompt tuning. Note that SPT is not baseline focusing on prompt transfer. Instead, it focuses on the prompt initialization strategy. Yet, attention transfer largely depends on the source prompt, whose ability on the target task further restricted by the modality and task gaps, which, do not appear in SPT. Therefore, attention transfer functions under a more challenging setup, but still achieves an average performance comparable to or even better than SPT. Given that SPT and attention transfer can be regarded as two different perspectives for prompt initialization, even though attention transfer falls below SPT on some tasks, the results still demonstrate the potential of cross- mods-mty prompt transfer.
CONCLUSIONS
In this paper, conducted extensive experiments to verify the fcasibility and study the prompt trans- ferability for cross- cross-modality prompt transfer. We also proposed a prompt transferability estimation method based on gap analysis. Lastly, further demonstrated the potential and effectiveness of cross-modality prompt transfer through our proposed attention transfer, boosting prompt tuning to a level that matches and even outperforms the newest prompt-based PETL benchmark. Our research opened new path to benefiting data-scarce tasks with the rich resources of the text modality, We hope our research can serve as solid foundation for the topic of cross-modality prompt transfer and provide new directions and inspirations to mitigating challenges faced by data-scare tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The work was supported by the Australian Research Council (ARC) under Laureate Fellow grant FL190100149.
REFERENCES
Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, and Luis Espinosa-Anke. Tweet- eval: Unified benchmark and comparative evaluation for tweet classification. arXiv preprint arXiv:2010.122 12421, 2020.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint ar0iv: 05326, 2015.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, (55(10):8855-18 2017.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efffiient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, ((3):220-233 2023.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop paraphrasing (IWP2005), 2005.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929, 2020.
Emma Dugas, Jorge Jared, and Will Cukierski. Diabetic retinopathy detection, 2015. URL https het.abeeee-c-eeetnnpapptht-eeeee
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012. Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf. and Alexander Smola. A
Guo, Boyang Li, and Han Yu. Improving the sample efficiency of prompt tuning with domain adaptation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3523-3537, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 204-207. IEEE, 2018.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. arXiv preprint 2xXi::0008002275 2020.
11

---

Published as a conference paper at ICLR 2025
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning, pp. 2790- 2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Ahue Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 09680, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Fei-Fei Li, Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(12)):11111111111 1938.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 141::9:412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. I1-104. IEEE, 2004.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint arXiv:2104.00 04660669, 2021.
Fei-Fei Li, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Xiang Lisa Li and Percy Liang. frefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv: 100000,,,,,, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXiv 11692, 2019.
Kevin Lu, Aditya Grover, Pieter Abbeel. and Igor Mordatch. Frozen pretrained transformers as universal computation engines. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 7628-7636, 2022.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://github.com/ hwwpe///owoeeppii/ppppptttteieee
2017.
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209. 15/62, 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep feature
learning and insupervised feature learning, volume 2011, pp. 7. Granada, Spain, 2011.
M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06) volume 2, pp. 1447-1454. IEEF, 2006.
12

---

Published as a conference paper at ICLR 2025
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arX:v:180..03748, 2018.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. -908-5 IEEE, 2012.
Bo Peng, Zhen Fang, Guangquan Zhang, and Jie Lu. Knowledge distillation with auxiliary variable. In Forty-first International Conference on Machine Learming, 2024.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammed AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. Semeval-201 task 5: Aspect based sentiment analysis. In ProWorkshop on Semantic Evaluation (SemEval-2016), pp. 19-30. Association for Computational Linguistics, 2016.
Lakshay Sharma, Laura Graesser, Nikita Nangia, and Utku Evci. Natural language understanding with the quora question pairs dataset. arXiv preprint
2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language pro- cessing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3949-3969, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology- In Medical Image Computing and Computer Assisted Intervention-MICCA. 2018: st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pp. 210 218. Springer, 2018.
Tu Vu, Brian Lester, Noah Constant, Rami AI-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 5039-5059, 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: (tl::/804.075 07461, 2018
Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, et al. Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9147-9160, 2023.
Ran Wang, Hua Zuo, Zhen Fang, and Jie Lu. Towards robustness prompt tuning with fully test-time adaptation for clip's zero-shot generalization. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8604-8612, 2024a.
Yuzhu Wang, Lechao Cheng, Chaowei Fang, Dingwen Zhang, Manni Duan, and Meng Wang. Revisiting the power of prompt for visual tuning. arXiv preprint arXiv:2402.02382, 2024b.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv: / 704.05522, 2017.
Jianxiong Xiao. James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and patlern recognition, pp. 3492-3492. IEFE, 2010.
13

---

Published as a conference paper at ICLR 2025
Omar F Zaidan, Jason Eisner, and Christine Piatko. Machine learning with annotator rationales to reduce annotation cost. In Proceedings of the NIPS 2008 workshop on cost sensitive learning, pp. 260-267, 2008.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]: Learning vs. learning to recall. arXiv preprint arXiv:2104.05240, 2021.
14

---

APPENDIX
Contents
Influential Prior Studies
Preliminaries
B. I Maximum Mean Discrepancy
B.2
Prompt Transterability Estimation: Model Stimulation Similarity
Technical Details
C.1
Image Processing and Augmentation
C.2
Training Prompts on NLP Tasks
C33
Transferring Prompts to CV Tasks
C.4
Cooss-Modality Prompt Transferability Estimation
C.4.1
Modality Gap)
C.4.2
Task Gap)
C,5
Per- Task Training Hyper- Parameters
C.5.1
Linear Probing
C.5.2
Vanilla Visual Prompt Tuning
C.5.3
Cross-Modality Prompt Transter
6
C.5.4
Training the Universal Projector for Measuring Task Gap
Detailed Results
D.I
Cross-Modality Prompt Transfer
8
D.1.1
Frozen Prompt Transter & Projection Transfer
D.1.2
Attention Transfer
8
D.2
Cross-Modality Prompt Transferability Estimation
D.2.1
Baseline Method: Cosine Similarity between Average Prompts
D.2.2
Baseline Method: ON
D.2.3
Baseline Method: Pretraining Data Volume
D.2.4
Modality and Task Gapl.
D.2.5
Ground- Truth Gap Values
10
Discussions
12

---

A INFLUENTIAL PRIOR STUDIES
Our work is built upon a series of prior studies, which offered great help to the conceptualization, implementation, and conclusion of our work. We list these influential prior studies in Table [ along with their contributions to our study. Here we wish to express our sincere thanks to the authors for their great and reproducible works, without which there will be no begins of work.
Table 1: Influential prior studies that are of great help to our work.
Authoris)
How they help research
Viual Tuning
2022
BCCV2022
The concctullaaiion
the publicly available waining recelpes VTAB
Senutivity Aware Ysurrrarnzer Emiclet
ICCV2023
The
avulibbl dasset
SPoll Better Frowen Model Adaptatio through Sun Trarch Va NO0 On Transferaiiityy of Prompe Tuning for Natual Langage Paazziig
AC1.5022
The ommpiuallaaion proupt tnndir
NAACL2302
The publicly availbll onder and tninne prints
B
PRELIMINARIES
B.I MAXIMUM MEAN DISCREPANCY
Maximum Mean Discrepancy (MMD) non- nnn-paramerri method used measure the difference between two probability distributions P and Q based on samples drawn from these distributions. The
MMD is particularly useful in two-sample hypothesis testing to determine whether / Q.
Given two sets of samples {II,..., Im drawn from distribution P and {.1,..., 3/ drawn from distribution Q, MMD defined as the distance between the mean embeddings of the two distributions in reproducing kernel Hilbert space (RKHS). Mathematically, the MMD is computed as:
MMD Q; [[][]())] EE [o)d())]
where è is a feature map into the RKHS H. Expanding the square norm gives the following expression:
(MD0(P,Q:70>> E2,z/
Ey. [kV, v/) Mu 2T v)
(2)
Here, k(x, y) is a positive-definie kernel function, such as the Gaussian or linear kernel, which defines the inner product in the RKHS.
Intuitively, the MMD measures how well the mean of the samples from P Pndd Q match in the RKHS. If the distributions are identical, the MMD will be zero.
B.2 PROMPT TRANSFERABILITY ESTIMATION: MODEL STIMULATION SIMILARITY
The prompt transferability estimation method proposed by Su et al. 2022) is chosen our baseline. However, due to page limitation, their method (named ON) not introduced in detail. Therefore in this section, we provide brief but detailed introduction on their proposed method.
measures the prompt transferability between two NLP tasks 81 and 82 by the following steps: (i) feed the prompt Ps, (trained on $1) solely to RoBERTa and record its activation map A1. The activation map of a single feed-forward layer is a binary vector obtained by setting the activation values greater than zero to and 0 otherwise. The final activation map is the concatenation of the single-layer activation maps the last three attention blocks. (ii) perform step (i) on the prompt Pxz (trained on sy) obtain its activation map Ag. (iii) calculate the cosine similarity between A₁ and Ag as the final transferability score between $1 and 82:
A x Ag ON Pe,) = || x ||22|
In our c--imllementatinn since the source prompts transferred to CV tasks and will be used pretrained ViT model, we use ViT instead of RoBERTa to calculate the model stimulation similarity between prompt and target prompt trained CV task.
2

---

C TECHNICAL DETAILS
C.I IMAGE PROCESSING AND AUGMENTATION
For the VTAB-IK IK ZZai et al. 2019) dataset, we use the scripts provided by He et al. (2023 to download and convert the dataset into PNG images. To load the images into our codes, we use the Dataset class provided by PyTorch (Paszke et al. 2019).
Regarding the image augmentation strategies for VTAB-IK, we follow the default settings in VTAB- IK and do not any augmentation tricks except the following:
1. Resizing the images to size of 224 224;
2. Converting the images to PyTorch tensors and re-scaling them to 0
3. Normalizing the images using the mean and standard deviation values calculated from ImageNet (mean (0.485, 0.456, 0.406), sld (0.229, 0.224, 0.225)).
C.2 TRAINING PROMPTS ON NLP TASKS
The source prompts trained with RoBERTa on different NLP tasks are directly adopted from the official code repository of Su et al. 2022 Below we show some crucial information for training the source prompts:
Input Embedding Structures. For NLP tasks that input one sentence at time (such as sentiment analysis), the input embedding structure in Table is adopted. While for NLP tasks that input two sentences at the same time (such natural language inference, ethical judgment, and paraphrase identification), the input embedding structure in Table Bs adopted.
In the tables, the top shows the order of different tokens. The Length row indicates the length of each token. The Learnable? row indicates whether the corresponding tokens are learnable or not. Finally, the Positional? row indicates whether the positional embeddings of the pretrained RoBERTa are added to the corresponding token.
Table 2: The input embedding structure for training prompts on single-sentence NLP tasks.
[MASK]
[Prompt]
[CLS]
[Input Sentence]
[SEP]
Length
99
Learnable?
V
V
X
Positional?
x
V
V
V
Table 3: The input embedding structure for training prompts on dual-sentence NLP tasks.
[MASK]
[Prompt]
[CLS]
[Input Sentence 1]
[SEP]
[Input Sentence 2] [SEP]
Length
99
Learnable?
V
V
x
x
X
x
x
Positional?
V
Hyperparameters. In fact, the hyperparameters used to train the prompts on different NLP tasks will be omitted here. The complete training recipes on each NLP task, including the hyperparameters. optimizer, learning rate scheduler, and can be found in the original code repository of et al. (2022). This link shall direct you to their training recipes.
https://gituub.oo///ziplbb////
3

---

C.3 TRANSFERRING PROMPTS TO CV TASKS
Input Embedding Structures. In all of our transfer settings, as long as there is prompt prepended to the image patch embeddings, no matter how the prompt is obtained, the prompt length will always be 100 and the input embedding structure demonstrated in Table D is adopted. This applies to not only cross-modality prompt transfer scenarios, but also to vanilla visual prompt tuning scenarios.
Table 4: The input embedding structure for VIT.
[Prompt]
[CLS] [Image Patch Embedding]
Length
100
196
Learnable?
X
Positional?
V
Frozen Prompt Transfer. Frozen Prompt Transfer simply prepends the trained but frozen source prompts the image patch embeddings and only learns an output predictor on top of the [CLS] features. The only thing that needs attention is the output predictor: it consists of a linear layer (with bias) whose input dimension is the hidden dimension VIT (768 in our case) and output dimension is the number of classes of the target CV task. The same output predictor is used across every settings in our paper.
Projection Transfer. The core module in projection transfer the linear projector, which is simple linear layer (with bias) with input dimension of 768 (hidden dimension of RoBERTa) and output dimension of 768 (hidden dimension of ViT).
C.4
PROMPT TRANSFERABILITY ESTIMATION
C.4.1 MODALITY GAP
In this paper, the modality gap between a source NLP task and target CV task is measured as the Maximum Mean Discrepancy (MMD, Gretton al. (2122) between source prompts and target image patch embeddings. This section will explain on how to calculate the MMD between a source prompt (100 vectors, each with dimension of 768) and the target image patch embeddings (1000 196 vectors, each with dimension of 768). Note that we use the training set of each target task for MMD caleulation. This means that for every target task, we only 1000 images for MMD calculation. Since an image will be converted into 196 patch embeddings, we would have 1000 x 196 vectors in each target CV task for MMD calculation. The procedure of measuring the modality gap is illustrated in Algorithm
Algorithm I Measuring modality gap via MMD
Input: Source prompt p. with dimension (1, dna); Vision task T;; ViT's image embedder E,, Output: Measured modalit gap between ps and Tr
1: Extract patch embeddings: è = (T)) (dimension: ([Tr), Np, dne). No is the # of patches)
2: Reshape e; to ( Ti x Np. dhe); Then shuffle e, along the 0-th axis
3: (Mooaal, Nsteps) = (0
4: for epoch in range(5) do
5: while is not sampled out do
6: Sample embeddings from e;, denoted é;
7: Calculate the MMD between P. and é;: Meurrent MM,,p,,é)
8: Miotal+= Meurrent
9: N
end while
11: end for
12: return Mootat/Neten
4

---

C.4.2 TASK GAP
Source Prompis: p?
Universal
Projected Procapt
Targel Prontg
IMDB
SST2
MRPC
Projector
Euclidean
Mean
Menn
Figure Training the universal projector for estimating the task gap.
In this paper, the task gap between a source NLP task and target CV task is measured the cosine similarity between the projected source prompt and the target prompt obtained through vanilla visual prompt tuning on the target CV task. The core module here is the universal projector that projects all the source prompts to the space of the target prompt. The universal projector itself is a simple linear projector (with bias) with an input dimension of 768 (the hidden dimension of RoBERTa) and output dimension of 768 (the hidden dimension of ViT). Given a target CV task, Figure demonstrates the procedure for training the universal prompt projector:
Step Perform vanilla visual prompt tuning on the target CV task with a pretrained ViT to obtain the target prompt Pt.
Step Shuffle all the source prompts trained on each NLP task and mix them with each other. In total there are 13 source tasks, which means that if we gather all the source prompts, we would have 13 x 100 vectors, each with a dimension of 768. To shuffle and mix them, we simply rearrange the (13, 100, 768) vectors into (1300, 768) vectors, shuffle them along the 0-th axis and rearrange them back to the shape of (13, 100, 768).
Step 2 Pass the chuffled-and-mixx- source prompts to the universal projector Pu. Let's denote the projected source prompts as Pu (pail, ), the universal projector Pa is updated by the Euclidean distance between Pa (p and ps:
arg min (Pu Ps
mean(p:)llz
In practical, mean (ve has a shape of (13, 768) while mean((:) has a shape of (1, 768), the final Euclidean distance is calculated by averaging the total Euclidean dis- tance obtained by summing up the distance between mean(p:) and the 13 vectors from mean( (pel)
After the universal projector is trained, the task gap between source task and the target CV task can be measured by the cosine similarity between the projected source prompt and the target prompt. The reason why the source prompts are shufiled and mixed is two-fold: (i) To avoid overfitting and reduce susceptibility on hyperparameters, as in practical, we will not have ground-truth results to tune the hyperparameters. (ii) To further boost the prompt transferability estimation performance. The effects of the shuffling and mixing operations will be demonstrated in Section C.5.4
C.5
PER-TASK TRAINING HYPER-PARAMENE
C.5.1 LINEAR PROBING
The complete set of hyperparameters for linear probing with respect to each target task can be viewed in Table 5 The total number of epochs 100. Note: (i) For linear probing, we did not perform parameer grid-search on any of the target tasks, the hyper-parameters in the table released by Jia et al. (202)) and we simply follow. (ii) The Base LR column indicates the base learning rate for each target task. This base learning rate will be scaled according to batch size: Final LR = (Base LR batch size) before the training starts. (iii) The linear probing performance reported in main paper obtained by our re-implementation, with is nearly identical to that reported by Jia et al. (2022). Therefore, we did not question this set of eeper-parameter at all. (iv) The set of hyper-paramen parammerrsssssss...e also applied frozen prompt transfer, regardless of the source
5

---

prompts used. It would be highly unrealistic for to perform grid-search on frozen prompt transfer whenever the source prompt is changed.
Table 5: Hyper parameters for linear probing and frozen prompt transfer.
Task
Optimizer
Momentum
Base LR
LR Warm-Up
1.R Decay
Weight Decay
Batch Size
Caltech101
SGD
0.9
2.5
10
Cosine
0,001
2048
CIFARI00
SGD
0.9
0.1
10
Cosine
0
2048
DTD
SGD
0.9
I
10
Cosine
0
2048
Flowers102
SGD
0.9
0.1
10
Cosine
0.001
2048
Pets
SGD
0.9
0.25
Cosine
0.001
2048
Sun397
SGD
0.9
0.5
Cosine
2048
SVHN
0.9
0.25
Cosine
0.01
2048
Patch Camelyon
SGD
0.9
0.08
Cosine
0.01
2048
Resise45
SGD
0.9
0.5
Cosine
2048
EuroSat
SGD
0.9
10
Cosine
0.0001
2048
Retinopathy
SGD
0.9
0.1
10
Cosine
0.01
2048
DMLab
SGD
0.9
0.5
10
Cosine
0
2048
Kitti
SGD
0.9
5
10
Cosine
0
2048
SmallNORB/azi
SGD
0.9
2.5
10
Cosine
0.01
2048
SmalINORRB/ele
SGD
0.9
2.5
10
Cosine
0.01
2048
dSprites/lcc
SGD
0.9
5
Cosine
0
2048
dSprites/ori
SGD
0.9
0.1
Cosine
2048
Clevr/dist
0.9
0.5
Cosine
0.001
Clevr/count
SGD
0.1
Cosine
2048
C.5.2 VANILLA VISUAL PROMPT TUNING
The complete of hyperaarameters for vanilla visual prompt tuning with respect each target task can be viewed in Table The total number of epochs is 100. Note: (i) The original VPT repository also has different prompt lengths on different target CV tasks, while in our experiments we fix the prompt length to 100 on every target task. Therefore, their hyperparameters may not be suitable for our re-implementation. The hyperparameters in the table are obtained by our grid-search results. (ii) However, the original VPT uses a prompt length of 100 on the following tasks: CIFAR 100, DMLab, Kitti, DSprites/loc, dSprites/ori, and Clevr/count. For these tasks, also perform grid search to find a presumably best combination of base learning rate and weight decay values. We later perform VPT using this set of values and compare the test accuracy with that obtained by the original set of hyperparameters. Finally, we take the set of hyperparameters that gives the best test accuracy. Among the tasks with prompt length of 100, CIFAR100, DMLab, Kitti, and dSprites/ori keep using the original set of hyperparameters. While DSprites/loc and Clevr/count use the set of hyperparameters grid-searched by (iii) Similar linear probing, the base learning rate will be scaled according to batch size: Final LR
carefully grid-searched hyperparameters for VPT. However, the original VPT adopt
(Base LR * batch size) /56, before the training starts.
C.5.3 CROSS-MODALITY PROMPT TRANSFER
For frozen prompt transfer, the set hyperparameters is exactly the same to linear probing regardless of the source prompt prepended, as depicted in Table The total number of epochs is also 100. is worth mentioning that for some of the target tasks (particulary SmallNORB/azi SmallNORB/ele, dSprites/loc, and dSprites/ori), specific source prompts (particularly MNLI, QNLI, SNLI, and QQP prompts that are performant in projection transfer). We suspect that the hyperparameters used in linear probing not the optimal hyperparameters for these transfer tasks. However, as mentioned before, its impossible for us the search for the best set of hyperparameters for every source-target pair, given the fact that there are 247 pairs in total. What competency of the source prompts. Therefore, we did not try to draw any other conclusions but "whether prompts trained NLP tasks be safely transferred to CV tasks" from the experimental results of frozen prompt transfer.
observed extremely big loss values 1000) when using some
hope universal set of hyperparameters that can highlight the relative
For projection transfer, the Adam (Kingma & Ba 2014) optimizer is used, with learning rate of 0.005, a weight decay of 0.001, and a batch size of 64. No warm up steps or learning rate scheduler are adopted. The total number of training epochs is also 100. This set of hyperparameters is grid-searched

---

Table 6: Hyper-parameters for vanilla visual prompt tuning.
Task
Optimizer
Momentum
Base LR
Warm-Up
Decay
Weight Decay
Batch Size
Caltech101
SGD
0.9
10
10
Cosine
0.001
64
CIFARI00
SGD
0.9
Cosine
0.001
DTD
0.9
Cosine
0.001
Flowers102
0.9
2.5
Cosine
0.001
Pets
SGD
0.9
2.5
Cosine
0.001
Sun397
SGD
0.9
Cosine
0.001
SVHN
SGD
0.9
0.5
10
Cosine
0.01
64
Patch Camelyon
SGD
0.9
0.5
10
Cosine
0.01
64
Resisc45
SGD
0.9
10
10
Cosine
0.001
64
EuroSat
SGD
0.9
5
10
Cosine
0.0001
64
Retinopathy
SGD
0.9
2.5
10
Cosine
0.01
64
DMLab
SGD
0.9
500
10
Cosine
0
64
Kitti
SGD
0.9
250
Cosine
SmalINORB/azi
0.9
Cosine
0.001
SmallNORB/ele
0.9
Cosine
0.001
dSprites/loc
SGD
0.5
Cosine
0.01
dSprites/ori
SGD
0.9
0.5
Cosine
0.01
Clevridist
SGD
0.9
0.1
Cosine
0.01
64
Clevr/count
SGD
0.9
0.5
10
Cosine
0.01
64
on some of the target tasks with a source prompt trained on IMDB, We found that multiple target tasks give similar grid-search results. Therefore we ceased searching on the rest of the tasks and extend the set of hyperparameters to every source-target pair.
C.5.4 TRAINING THE UNIVERSAL PROJECTOR FOR MEASURING TASK GAP
Given target CV task, to measure the task gap between it and all the source NLP tasks, the universal projector needs to be trained. The function of the universal projector to project all the source prompts to the space of target prompt. It is optimized by an Adam optimizer with the following hyperparameters: learning rate of 1e-4, a weight decay of 0.1, batch size prompts are passed to the universal projector at once), with warm up steps and no learning rate scheduler. This set of hyperparameters is grid-searched on CIFARIO0 and extended as-is to all of the target tasks. Specifically, after training the universal projector with LR-WD value pair, evaluate the Kendall's coefficient between the ground-truth projection transfer rank and the rank of cosine-similand between the target prompt and the projected source prompts. We train the universal projector on three different random seeds: [42, 44, 100] and report the grid-search results in Table 7
(all source
Table Grid-search results on CIFARI00 for the hyperparameters of the universal projector.
JLR WD->
0
0.1
0.01
0.001
0.0001
0.001
52.14+10.33
36.75+1.2. 211
36.75+1.22 1.22
50.43+6.04
52.14+10.33
0.0005
80.3448.46
45.30+1.21
73.50+8.46
80.34+8.46
80.3448 46
0.0001
81.20#4.36
82.91+1.21
81.20+3.20
82.05+4.19
81.20+4.33
0.00005
76.92+0.00
75.21+1.21
76.9240.00
76.92+0.00
76.92+0.00
0.00001
50.43+14.85
38.46426.73
50.43+14.85
50.43+11.85
50.43+14.85
When the learning rate is set as 0.0001 and weight decay is set as 0.1, the universal projector gives the best cross cross-modality prompt transferability ranking score. Therefore, this set of hyperparameters is used on every target task. We need to emphasize that although performing grid-search on every target task would give better ranking scores, it is not right to do that every task. As in practical, we need to know the transferability ranking scores before knowing the ground -trutt transfer performance. Using the ground-truthe transfer performance as the reference for conducting grid-search is paradoxical in practical scenarios: if we already have the ground-truti transfer performance, why would we even need estimate the transferability.
7

---

D DETAILED RESULTS
D.1 CROSS-MODALITY PROMPT TRANSFER
D.1.1 FROZEN PROMPT TRANSFER & PROJECTION TRANSFER
The detailed results in Table 14 Note that all source-target pairs three random seeds: [42, 44, 100], and their best results are reported in the format meantstd.
D.1.2 ATTENTION TRANSFER
For attention transfer, release the training recipe and best accuracy in the format of meantstd on each target CV task in Table Similarly, all experiments are ran on three random seeds: [42, 44, 100).
Table 8: Hyper-parameter for attention transfer. The Source column shows the source prompt used for transfer while I, indicates the concentrated source prompt length. Note that on each target task, the Adam optimizer with batch size of 64 is adopted. The target prompt length is always set as (100 - L.) on every target task.
Task
Source
LR
LR Warm- Up
LR Decay
Weight Decay
Accuracy
Caltech101
SNLI
80
0.005
0.001
90.09=0.28
CIFARI00
QQP
60
0.005
0,001
81.7220.34
DTD
MNLI
0.005
0,001
68.9240.24
Flowers102
MNLI
60
0.01
10
Cosine Cosine
0.001
98.3720.12
Pets
MNLI
90
0.01
10
0.001
89.42=0,00 008
Sun397
SNLI
50
0.005
0.001
52.57+0.31
SVHN
SNLI
60
0.01
10
Cosine
0.001
82.76+0.24
Patch Camelyon
SNLI
80
0.1
10
Cosine
0.0001
82.08+1.44
Resisc45
MNLI
100
0.01
Cosine
0.001
82.92+0.33
EuroSat
SNLI
100
0.01
10
Cosine
0.001
96.17+0.11
Retinopathy
QNLI
20
10
Cosine
75.30#0.44
DMLab
QQP
80
0.005
10
Cosine
0.01
44.85+0.31
Kitti
QNLI
70
0.005
0.001
77.37+0.40
BmallNORBBari
QQP
100
0.001
10
Cosine
0.0001
24.34+1.88 1.82
Beele
SNLI
100
0.0005
10
Cosine
36.26*1.00
dSprites/loc
SNLI
0.005
Cosine
73.1340.40
dSprites/ori
SNLI
90
0.005
10
Cosine
0.001
43.16+1.66 1.6
Clevr/dist
MNLI
90
0.001
10
Cosine
60.75+0024
Clevr/count
SNLI
100
0.001
10
Cosine
67.9240.51
D.2
CROSS-MODALITY PROMPT TRANSFERARILLITY ESTIMATION
D.2.1
BASELINE METHOD: COSINE SIMILARITY BETWEEN AVERAGE PROMPTS
We show the best (SmallNORB/azi) and worst (Kitti) ranking results of the baseline method: Avg Cos in T
D.2.2 BASELINE METHOD: ON
Similarly, the best (Flowers102) and worst (Resisc45) ranking results of ON are shown in Table I0
D.2.3 BASELINE METHOD: PRETRAINING DATA VOLUME
For this method, we simply list the volume of pretraining data of each source task in Table |I
D.2.4 MODALITY AND TASK GAP
The best (Clevr/count) and worst (Retinopathy) ranking results shown in Table 12
8

---

Table 9: The cross-modality prompt transferability ranking results of Avg Cos SmalINORB/azi (left) and Kitti (Right), Score corresponding transferability score of each source prompt, predicted by the prompt transferability estimation method.
SmallNORR/ari
Kitti
Rank
Estimated
T Score
Ground Truth
Accuracy
Estimated
T Score
Ground Truth
Accuracy
sst2
0.042
qqº
23.61
laptop
0.033
quli
76.98
tweet
0.040
quli
23.36
deontology
0.027
mnli
76.84
quli
0.039
sst2
22.72
movie
0.010
76.56
mipe
0.037
deontology
22.51
imdb
0.007
snli
76.47
mnli
0.036
snli
21.72
restaurant
0.001
sst2
75.86
6
qqp
0.033
tweet
21.63
ss12
-0.001
tweet
75.25
deontology
0.029
imdl
21.50
mipe
-0.006
deontology
74.31
justice
0.006
mnli
21.05
justice
-0.014
justice
72.53
imdb
-0.002
justice
20.74
snli -0.032
mrpe
71.73
10
snli -0.003
mrpe
19.80
mnli
-0.035
imdh
71.68
11
laptop -0.019
restaurant
18.62
-0.039
restaurant
69.81
12
restaurant
-0.046
laptop
16.82
tweet
-0.046
movie
66.62
13
movie -0.049
movie
16.35
gnli
-0.077
laptop
65.92
Table 10: The cross- prompt transferability ranking results of ON on Flowers 102 (left) and Resise45 (Right).
Flowers102
Resise45
Rank
Estimated
T Score
Ground Truth
Accuracy
Estimated
T Score
Ground Truth
Accuracy
snli
0.429
mnli
98.22
deontology laptop
0.448
snli
82.01
qqp
qqp
98.20
0.392
mnli
81.92
qnli
0.407
snli
98.19
movie
0.382
81.54
mnli
0.390
quli
98.04
mrpe
0.373
qnli
81.
sst2
0.376
imdb
97.99
tweet
0.366
sst2
80.86
imdh
0.372
97.94
sst2
0.359
tweet
80.07
deontology
0.364
sst2
97.93
justice
0.340
imdb 79.63
tweet
0.352
justice
97.82
qnli
0.312
justice
79.04
justice
0.329
deontology
97.66
imdb
0.312
deontology
78.08
10
movie
0.328
restaurant
97.69
restaurant
0.307
mrpc 77.00
11
laptop
0.311
mrpe
97.59
qqp
0.303
movie 76.95
12
mrpe
0.299
laptop
97.48
mnli
0.283
restaurant
76.78
13
restaurant
0.292
movie
97.45
suli
0.278
laptop
76.
Table 11: Pretraining data volume of each source NLP task.
Source Task
imdo
s812
laptop
restaurant
movic
tweet
muli
qali
snli
doontology
justice
Data Volume
25000
67349
3041
1600
45389
392702
104743
549367
18164
21791
363846
Table 12: The cross-modality prompt transferability ranking results our method (G & GT) on Clevr/count (left) and Retinopathy (Right).
Clevr/count
Retinopathy
Rank
Estimated
T Score
Ground Truth
Accuracy
Estimated
T Score
Ground Truth
Accuracy
snli
-0.619
snli
62.60
-0.213
quli
74.76
mnli
-0.675
mnli
58.52
snli
-0.269
qqp
74.76
-0.698
qnli
57.67
qnli
-0.311
snli
74.55
quli
-0.744
qqp
56,14
mnli
-0.313
sst2
74.53
sst2
-0.776
sst2
55.24
ss12
-0.659
mnli
74.49
tweet
-0.980
tweet
55.01
tweet
-0.220
tweet
74.42
deontology
-1.021
deontology
53.02
deontology
-0,935
laptop
74.40
justice
-1.291
imdb
52.77
justice
-1.122
dcontology
74.37
imdh
-1.355
justice
52.27
imdh
-1.144
movie
74.37
10
mipe
-1.463
mrpc
51.75
mipe
-1.258
imdb
74.36
11
restaurant
-1.844
restaurant
48.72
restaurant
-1.425
mipe
74.29
12
laptop
-1.916
laptop
45.34
laptop
-1.460
justice
74.24
13
movie
-1.962
movie
44.38
movie
-1.476
restaurant
74.24

---

D.2.5 GROUND-TRUTH GAP VALUES
The ground-truth values of the combined modality and task gaps are in Table 13
Table 13:
The ground-truth values of the combined modality and task gap.
IMDB
SST2
Laptop
Restaurant
Movie
Tweet
MNLI
QNLI
SNLI
Deontology
Justice
QQP
MRPC
Calteh1111
1.320
CIFAR100
1455
0.721
1.958
0.118
0.077
1.553
DID
1.344
1.061
1.472
1/461
1.485
1.179
0.101
0.538
0.063
1.219
1.311
0.393
1.375
Flowens102
1.412
1,094
1.561
1.551
1.579
1.224
0.128
0.561
0.127
1,300
1.390
0.374
1,450
S97
1,286
0.881
1.454
1.440
1.488
1,050
-0.073
0.347
-0.109
1.176
1.283
0.124
1.329
SVHN
1,640
0,801
2,599
2,462
2,650
1,032
0.663
0.591
0.657
1.214
1.546
0,671
1,859
Camelyon
Resise45
1,310
0.836
1.553
1.530
1.583
0.974
0.003
0.223
-0.032
1.125
1.287
0.074
1.364
EuroSAT
1,253
0.873
2,525
2,415
2.565
1,160
0.396
0.447
0.393
1.309
1,660
0,464
1.972
Reiinopathy
DMLab
0.924
1.152
-0.045
-0.463
0.752
-0.271
KITTI
0.479
0.214
0.713
0.639
0.574
0.301
-0.608
-0.187
-0.602
0.377
0.170
-0.390
0.512
SmaliNORB/ari
2.784
1,870
3.545
3,466
3,607
2.132
0.721
1.185
0.715
2.307
2.697
1,084
2.951
SmallNORB/ele
dSpritessloc
3.445
2,750
3.907
3,853
3,930
3
1.128
2,024
1.127
3.125
3.428
1,766
3.586
USpritestori
3.439
2.725
3,895
3,838
3.922
2.997
1.107
1.998
1,152
3,144
3,420
1,707
3.568
Clevidist
Clevnlcount
1.355
0.776
1.916
1844
1.962
0.980
0.675
0.744
0.619
1,021
1.291
0.698
1,463
10

---

Table 14: The detailed results of Frozen Prompt Transfer (gray) and Projection Transfer (Blue). The accuracy marked in red indicates the best source prompts for each target task.
Pach Cunebyon
Calasrinn
CIRAR000
Clovellel
1013
SVERN
Mon
Lawe:
65.25890.6
64,9280033
97,27=0001
86.55e0000
36.25e11.1 51.22
75/0240.36
69.54g0100
32.34
74
33.8380020
60.105544
11,92=0077
20.9000.3
20.97.0.01 12.07
26 26.2990004
Kaster
82 334000.7
61 89696.00 89
U5. 37.5169437
84 16666 3M
45 98681-1
37 77ktat111 18
06:00
W0 20:3443:301
06.63st1
S0 T1:a1 76
74. 15
77.80
51 31/994885
59.56401
(1.66401 17
18 18:0-4177 73
10 10:940 37
15 15M446111 19
531140031 31
37 37.4540:07
37.00
IMD8
$2.7500.13
62 323388002
39822306
95.015005
84.22a0 10
453445111 13
37.3120.09 09
67.01
79.7349.11
90.52-054 67.1520000
S4
74, 01 m1
77.85
33. 665.22
$8.0080 20
11,55en 23
19 56
11 110000
16 164120.01
3L.640.57
37.1223.06
21.45
S312
03.
858304
60.4433112
060092003
64-7360.00
37.18a1:0
79:122300
69
13.999116 16
79.09
33.5445.10
599220
20.0120
157220002
35-9333377
27.70
Lapop
05.57=000
84.726000
37. 7.5080007
80.33
50.52-151
74 74.050002
59.88e1.58
11.8e150
20 20.66000
9.57
15.77
52.0500.55
57 57.4240.06
27.5
Mtrii
52 12.2500055
62 346360
99618534
05 802302
84.3256 00
37. 37.56g
66.33
90.2563112 12
06.71231
13 15
77.79
53.4 41
61138633
115 16
19.961 16
1041
16 17.17
$1.3085500
37. 37.35
25:35
Deaa
65.06.0.00
62.810001
201002
84
10 m
11:32:1:11
82:51
60.58.99
24. 10000
78.10
SOL0012
14.10
17.17.0013
56 36.1110:0
28.30
MNLEI
56.00.3
200111
31 11-5-0--4
89.11
34.800 13
0.9000.2
18.77-041
12.110011
1564.0.01
ONLI
00022200
85 1l
37 17
79
1957
64 75
06.500000
51.310.15
11.88a0
315330.3
27.
Demnslogy
82 12.970..5
62.21150 2110.0
59.312500
05 0554
S3.35a 10
67.25
75.000115
76
64
90.34.015 15
74, 103602
11.90
33 33.513055
60006133
1145 06
19 19.880
13.0050.3
16 163838011
521898355
38.06
Jaciic
K2 2740
67 5246040
5000-000
85 1515500
83 03372411
15 15:304005
N3555000
67335-0 m
5000-115
11 W----
5166
51 51066415 1N
19 15-57-145 (106610
15.41-133 21
10 10904002
11806 370
57 571770001
7%. 12
OOF
649932006
02.3223-9
95455001
5509251
In
36.302000
08.80
79.6522115
714122006
06
90.52003
26
24.003113 13
7554
24011001
56112115
19.72000 6
10.5220.10
15 150055.0
32012035
357089-11
25.14
MRFC
$2.9120.13
62.4983008
S8,4833155
00,055005
94.460
19
35.0kan 45.-2111
78.868911
66.1760 10
88.90u
73/Woo111
33.4660.15
19:5
11.120004
16.720000
31.770.81
27-3063006
27.06
VPT
NK 77
181140.11
05.7140111
07.754011
50719411
71 117661 15
77.09
41-90-1147
73 75.450007
00.35
20.1111 2M
22 32.0540 11.60-11
72
30 34 25
45.05
Xasier
74.2810.57
00.51000
$1000
01.35
04
23.2100
62:25:1-1
530400.15
IMDS
14.1500:24
649760331
07.554000
75 2.00ll13
90
$6.11:0055
74 14.5e00
31.861
24.57
52.770057
S512
58 4.00e 05
7455260.5 15
06 06:005000
UT 073805000
56 35554 17
SE
75 16.35a1555
76.75
35
40
55 55.22-1 23
52.50
41 555551
15 36
1377 56
26.12a1
10 75
50 74
21
55 553440.5
45.25
Laplop
67.7850.14
73.020073
64.442015
07
50611016
75.30
91.32-043 76.1620000
74.40 11
00.82
37.2233042
6592966
23.08:0
57.901333
453411.50
41.29
Rostsrne
580740.17
73
61
51 $11 10
70.77
17
$1.46-11 m0
50.45
0211411 TM
057224.71
52.15an.1
76.35en.77
80.1720771
90.020045
55.17an03 31.322955
74.42-0.1
41.5340.54
75.2561:38
11.3an
26.26.0
63.9122.16
56.55a1.11
550122.99
MNLI
58.0
09.7700.0
50.9959.12
68.485.14
06 6C332302
59.022000 02
$2.0421166
70.53
90:113344
95.5kam
10
74. 74.4ton 20
83.25
43.5 43.3720092
76.3481335
26.4020 13660001
21,00am0
70.53e1.77 77
41 11.7221 34
58.4485555
54.5200.55
40.00
ONI
58
00.1300111 16
54-4-0033
$2180151
11:0-1:0
79.15
76-751865
81.66-171
05.17 106
74
14.6611:1
10.05
41.451175
76.9000
16 16.55-087
66. 1177--1-5
41 55-00.37
50 07-21:0 55
476710.17
49:00
SNLJ
50.7750.19 19
08.3143:22
85-0220111 19
522222115
51 7355
79.97
42.812112 12
95.752004
74.50l130 30
33:23
43. 15
764721/10
217224173
26512008 96
72. A4
42 12.3500.41
57.0
62.60
50.43
Deonology
38.2043.36
(5.3243.35
07 0728920007
218111111 11
16.0011:55
796169356
94.121122 22
74.371107
51.52
74.31=1166
26.0ke1
63.33 63.31=335
11:33
50.3641.35
53:113301
76.0411422
OGF
(0
07915513
30
88.4
51.77
62 3.45c1.11
29.64
W6
01.54
55.5
74
85.31
43.610024
76.6661
13.6=1155 157
16
7U.73l
40 103552.00
57.5760996
%6.142.44
MKPC
58 3436651.77
76.9269.31
31
646165.35
07. 59
S7.678000 07
50,5 56
13, M45l
77
75.13+131
77.00e91
23.55-123
74 74.251056
90.99
34,5961.16
71.73601
19.0
34.7291
(6 (0.3gg) 98
37 979911 24
55 55,0900336
51.7562.33
45.81

---

E DISCUSSIONS
Discussion on our choice of VIT pretrained checkpoint. We discuss our choice on the supervised ImageNet-2IK ViT checkpoint (ViT-IN2IK for short) from two perspectives:
Why not checkpoint pretrained objective: we need to demonstrate how significantly can cross-modalit prompt transfer boost prompt tuning. In the meantime, visual prompt tuning was initially experimented on the ViT-IN2Ik. Therefore, to make the comparison meaningful, we select the ViT-IN21k checkpoint.
different dataset or by a different pretraining
Why not language-aligned ViT checkpoint: we need first demonstrate that model, even pretrained without any textual knowledge (e.g. the ViT-IN21K checkpoint), can benefit from transferring prompts pretrained on pure textual data. Second, the source prompts are trained with language models on language tasks. The language models are not aligned with any variations of ViT checkpoint and the language tasks are not aligned with any image tasks. We believe that a language-alignas where there are connections across modalities. However in our cases, the two modalities are isolated. Therefore, we choose the ViT-IN21k checkpoint instead of language-aligned ViT checkpoint.
ViT checkpoint should be applied to situation
Real-world applications of cross-modality prompt transfer. In terms of real-world scenarios, cross-modality prompt transfer can be of help from two aspects: (i) improve the prompt tuning performance on data-scarce tasks or privacy sensitive tasks (for example, medical diagnosis or business analysis), as it transfers abundant source knowledge without depending on the source data. (ii) improve the prompt tuning performance on tasks that would benefit from combining knowledge across text and vision domains, especially when one modality has abundant labelled data provides complementary insights. For example, it can enhance medical image analysis by transferring prompts pretrained on text-based medical knowledge, support vision systems used in autonomous vehicles by transferring prompts trained on traffic rule texts, etc.
Why would the source prompts pretrained on NLP tasks transferable to other modality tasks? We analysis this problem through two perspectives: the source prompts themselves, and the language model used to produce the source prompts. (i) Transferable source prompts are typically pretrained on huge volume of NLP data. As result, these prompts should contain valuable semantics that are high-level enough to be extendable to other modalities. Moreover, they should also have generalizability powerful enough to generalize to unseen modality. (ii) On the other hand, language models trained on large and diverse datasets, learn representations that capture concepts, relationships, and structures beyond text alone. These abstract features may generalize well to other modalities, where the prompts can act as conceptual "anchors" that "carry" the representations across modalities.
Which target tasks are more likely to benefit from cross-modality prompt transfer? In our main paper, Section 4.2, we draw a conclusion that "Prompt tuning performance on target tasks that already benefit lot from prompt tuning is likely to be further boosted by cross-modality prompt transfer". Here we discuss the intuition behind this conclusion: If a target task benefits a lot from prompt tuning (i.e. has higher prompt tuning performance than linear probing), this indicates that this target task requires the pretrained model to learn new features, because simple linear probing on its original features does not help much. In another word, such target task would require the pretrained model to learn extra knowledge. Therefore, why cross- modalitty prompt transfer excels on such target task is easy to understand: it introduces more knowledge to the target task than vanilla prompt tuning. As vanilla prompt tuning only learns target knowledge while cross-modality prompt transfer learns both the source and target knowledge. In conclusion, if target task requires the pretrained model to grasp a lot of new features, then it would be more transferable target task, and vice versa.
The improvements made by projection transfer is proven to be statistically significant. We prove this statistical significance following the below steps: (i) Define our null hypothesis: the improvements are not introduced by transferring the source prompts. (ii) We choose one-sided test as
12

---

the effects will be in one direction (i.e. contribute the increase of the improvements) (iii) Collect data: Group (Conducting projection transfer on all 19 target tasks with a randomly initialized prompt): In this case, we have only one "source" prompt and 19 target tasks, which makes the total number of observations to be 19. Among these 19 observations, only 5 show improvements over vanilla prompt tuning, resulting in a percentage of 26.32%% (The data comes from the first row of Figure 3b.2) Group 2 (Conducting projection transfer on all 19 target tasks with all 13 source tasks); In this case, we have 13 source prompts and 19 target tasks, resulting in total number of 19 13=247 observations. Among these 247 observations, 158 show improvements over vanilla prompt tuning, with percentage of 63.97%. (The data comes from the remaining rows of Figure 3b.2) (iv) Calculate p-value: The final p-value is calculated to be 0.02%. That is, given the results we observed, there's s only chance of 0.02% that the improvements are not introduced by cross-modality prompt transfer. In another word, we are 99.98% certain that the improvements are statistical significant: they are caused by cross-modality prompt transfer, instead of error or random chances.
Limitations and future works. Our work contributes the field by the verification of the feasibility of cross-modality prompt transfer, the conceptualization of the transferability estimation metric and more effective cross-modality prompt transfer method. Besides its contributions, it also comes with the following limitations in the current stage: (i) The scope: only one model from each modality was involved in our experiments. Moreover, different prompt tuning variants (i.e. prefix-tuning! were not covered in our explorations. (ii) The dependence on trainable parameter of the proposed transferability estimation metric: this dependence could slightly increase the computation complexity of the metric. Given the current development of prompt transfer and the fact that we are the first work on the topic cross- modoltyy prompt transfer, these limitations are hard to avoid in the current stage. However, are confident that these limitations will not affect the conclusions drawn in this paper.
In the future, besides breaking the limitations discussed above (i.e. extending the scope and reducing the complexity of the transferability estimation metric), it would be interesting explore multi-source prompt fusion methods for pross-modality prompt transfer: how to fuse multiple text-pretrained prompts to make better utilization of the linguistic knowledge stored in the prompts. Our future work aims to further solidify our findings about cross cross-modality prompt transfer and further release the powers of the source prompts on data-scarce modalities.
13

---

REFERENCES
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf. and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter-effiiient finent In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 11825-11855, 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, pp. 709-727. Springer, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412. 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language pro- cessing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 9949-9969, 2022.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), pp. 5039-5059, 2022.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: arXiv:1910.00 191004467, 2019.
14


Reviews:
[{"review_id": "0Qe4K5VHHq", "reviewer": "Reviewer_4VnN", "summary": "This paper explores several aspects of prompt tuning in cross-modality settings (e.g., vision and language), focusing on feasibility, transferability estimation, and performance improvement. Specifically, the authors validate that source prompts pretrained on text data can be safely transferred to tasks in a different modality. They also propose a better metric for transferability estimation, which takes into account both modality and task gaps. Finally, the paper introduces attention transfer techniques for cross-modality prompt transfer, which are shown to perform comparably to single-modality prompt tuning algorithms such as SPT.", "questions": "1. What is the training cost associated with the proposed transferability metric? How does it scale with the number of tasks or modalities?\n2. What is the distribution of modality and task gaps in practice? Is there a recommended threshold to select the source prompt for transfer, instead of choosing the best?\n3. Could the paper provide more concrete evidence of improvements in data-scarce tasks using the proposed cross-modality transfer approach?\n4. Is there a way to make the transferability metric more efficient, reducing the need for the universal projector?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The motivation is good.\n* The overall logic and flow of the paper are clear.\n* The experiments are thorough, with a detailed exploration of various aspects of cross-modality prompt tuning.", "weaknesses": "* Although the paper claims that cross-modality prompt transfer can boost performance in data-scarce tasks, this is not convincingly demonstrated in the experiments, particularly when comparing to the SPT baseline.\n* The problem itself is interesting, and some findings, such as the proposed transferability estimation metric and the connection between single-modality and cross-modality prompt tuning, are valuable. However, the advantage of cross-modality prompt tuning over single-modality methods is not convincingly established.\n* The proposed transferability metric seems somewhat complex. It requires training a universal projector across tasks, which may introduce additional training costs and overhead.\n* The experiments lack strong evidence to show a clear advantage of cross-modality transfer, especially in data-scarce settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "819RB9DEiw", "reviewer": "Reviewer_mrx7", "summary": "This paper explores cross-modality prompt transfer, aiming to improve the performance of prompt-tuning methods when transferring prompts across different data modalities, specifically from NLP to CV tasks. The authors propose a novel approach using attention transfer and modality gap analysis to address data scarcity challenges in the target domain. Experiments on a wide variety of tasks demonstrate the feasibility and advantages of this cross-modality prompt transfer, highlighting the potential of leveraging rich source prompts for enhancing performance in data-limited scenarios.", "questions": "Although the paper provides theoretical and experimental support, practical applications, particularly real-world tasks where cross-modality transfer excels, are not thoroughly discussed. Besides, the method may face scalability issues when applied to larger models or when requiring complex attention transfer setups, which could limit its applicability in more resource-intensive settings .", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-organized and clearly written.\n- The paper introduces a new perspective in prompt transfer by exploring cross-modal prompt adaptation\n- The authors conduct a thorough analysis of the modality and task gaps, which improves the understanding of transferability in cross-modal scenarios, offering valuable insights for future research .\n- Attention transfer is proposed to mitigate modality and task gaps, offering an effective way to reuse and adapt prompts across modalities. \n- The paper provides substantial experimental evidence on multiple datasets, showing that cross-modal prompt transfer can yield performance comparable to, or even surpassing, recent prompt-based benchmarks .", "weaknesses": "- The paper would benefit from additional comparisons with same-modality prompt transfer baselines, which would help quantify the specific impact of cross-modal transfer approaches.\n- The approach seems sensitive to hyperparameters, which could affect reproducibility and performance stability, particularly in diverse target environments.\n- There are instances where cross-modal transfer underperforms or fails, the approach's limitations should be further discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hUfo6d09Gd", "reviewer": "Reviewer_v8xH", "summary": "- Prompt tuning has become a popular paradigm to adapt frozen models to new tasks. \n- While prompt-transfer within the same modality has shown to been helpful, the paper argues that this area has not been thoroughly explored. \n- The paper presents an analysis on feasibility of cross-model prompt transfer which is especially relevant for data scarce modalities. The paper studies various design choices through experiments and analysis.\n- L205-208: To do so, the problem is split into modality and task-based gaps and these are quantified. A new approach called attention transfer is introduced to reduce this gap which helps in improving the benchmarks. One should not have to go to the appendix to understand the intuitions behind the approach.", "questions": "Clarity: \n- L106-107: Unclear from context. \n- The discussion on the hypothesis on why cross-modal prompt transfer should even work must be discussed early to set stage for the problem and the proposed solution \n- Is this a key contribution of the paper - if so this must be discussed in the main paper rather than the appendix. \n- L269: wouldn't eq (8) increase the prompt size to > 100 ?\n- How is \"relative performance\" defined? Given how dense the figure is, authors must include conclusion of this experiment in the caption. \n- Statistical significance: Are the improvements statistically significant ? I see that the standard deviations are reported in the appendix. Authors should include a discussion on significance of improvements in the main paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes an interesting idea of transferring language prompts to vision for various data-scarce tasks. The idea is interesting and seems novel.\n- The paper is easy to follow for the most part. \n- Experiments are thorough. Many different ablations are presented, various design choices are explored. \n- The results seem promising and show benefit of using the approach.", "weaknesses": "While the overall idea, experiments and findings could be helpful to the community, I have the following concerns with this paper: \n\n- It is unclear why you want to quantify prompt transferability (equation 6) given it is a function of hyperparameters, training data etc. It would be helpful to provide some intuitions. \n- L274: What was the motivation behind chosing a IN-21k pre-trained checkpoint chosen ? Why not 1k ? Why not a language aligned model like CLIP ?  \n- Intuitions: Maybe I missed this in the paper, but a discussion on why should prompts be transferrable, what is the composition of the dataset where they transfer and where they don't will be very helpful to the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MfFiEynBFB", "reviewer": "Reviewer_Ve1v", "summary": "This work proposes to improve the performance of prompt tuning by exploring cross-modality prompt transfer, that is leveraging prompts pretrained on text modality to improve performance on few-shot image recognition. They first validate the effectiveness of cross-modality prompt transfer by empirical comparison between VPT, linear probing, frozen prompt transfer and projection transfer. They also propose an effective metric to estimate prompt transferability by combining modality gap and task gap scores. Finally, they propose the attention transfer which introduces the knowledge from source domain by an attention block. Experiments on VTAB-1K validate that attention transfer can improve existing SOTA method.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Most parts of the paper are clearly presented.\n2. Promising results with extensive experiments.\n3. Novel idea of cross-modality transfer instead of the intra-modality transfer.", "weaknesses": "My concern focuses on the effectiveness of the proposed method. (1) The proposed projection transfer and attention transfer have introduced extra parameters, the effect from extra parameters and source domain are not decomposed in the experiments in Fig.3. This makes it hard to judge if the source domain knowledge is really useful. The authors can further strengthen the experiments by 1) using random source prompt in projection transfer and attention transfer, which will make the number of learnable parameters match to that in experiments using source prompts from NLP domain, 2) using a source domain with many training data and gradually increasing the number of source training samples and inspecting if the source prompts learned with different number of source samples lead to increasing trend of improvement in projection transfer and attention transfer. (2) The effectiveness should be demonstrated using different language models, so as to validate the generalization ability. (3) The effectiveness should be demonstrated using different visual prompt tuning methods, including shallow and deep methods. (4) The accuracy on Natural subset decreases, making it hard to determine the effectiveness. Experiments on more datasets should be conducted to support the claim of this paper.\n\nBesides, the meaning of designing a good metric estimating prompt transferability should be also clarified. GM & GT outperforms Vdata by 3 points with respect to the mean Kendall's coefficient. To what degree does such an improvement contribute to enhancing the transfer performance on VTAB-1k? The work \"Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, Dongfang Liu: APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models. EMNLP 2023: 9147-9160\" also introduced attention into prompt tuning, the relationship of the proposed method to it should be also clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)