Title: Autonomous Knowledge Integration Enables Efficient Cross-Modality Prompt Transfer
Primary area: foundation or frontier models, including LLMs
Keywords: Prompt Transfer

Abstract: Cross-Modality Prompt Transfer (CMPT), which transfers prompts pretrained on a data-rich modality (e.g., text), seeks to address initialization instability and data scarcity in prompt tuning of pretrained transformers for tasks with limited data.
However, prior research in CMPT typically transfers all source prompt vectors without accounting for vector redundancy or incompatibilities.
This indiscriminate transfer may include irrelevant or even detrimental vectors, thereby negatively impacting the performance of the target task.
To mitigate this issue, we propose Selective Cross-Modality Prompt Transfer (S-CMPT), a method that automatically identifies and transfers only the most relevant prompt vectors.
S-CMPT employs a lightweight attention mechanism to select vectors that are most pertinent to the target task.
Furthermore, we introduce a simple regularization term to encourage the selection of diverse and non-redundant vectors.
The selected vectors are subsequently adapted to the target task through a linear projection layer.
As a result, S-CMPT achieves impressive simplicity and effectiveness, demonstrating significant accuracy improvements (+4.42\% over prompt tuning and +0.65\% over state-of-the-art prompt-based methods) while utilizing far fewer vectors, offering an efficient solution for data-scarce prompt tuning applications.