ICML 2025

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Chengyi Cai * 1 Zesheng Ye * 1 Lei Feng 2 3 Jianzhong Qi 1 Feng Liu 1
Abstract
Model reprogramming adapts pretrained models
to downstream tasks by modifying only the in-
put and output spaces. Visual reprogramming
(VR) is one instance for vision tasks that adds
a trainable noise pattern (i.e., a visual prompt)
to input images to facilitate downstream classi-
fication. The existing VR approaches for CLIP
train a single visual prompt using all descriptions
of different downstream classes. However, the
limited learning capacity may result in (1) a fail-
ure to capture diverse aspects of the descriptions
(e.g., shape, color, and texture), and (2) a possi-
ble bias toward less informative attributes that do
not help distinguish between classes. In this pa-
per, we introduce a decoupling-and-reweighting
framework. Our decoupled visual prompts (DVP)
are optimized using descriptions grouped by ex-
plicit causes (DVP-cse) or unsupervised clusters
(DVP-cls). Then, we integrate the outputs of
these visual prompts with a probabilistic reweight-
ing matrix (PRM) that measures their contribu-
tions to each downstream class. Theoretically,
DVP lowers the empirical risk bound. Exper-
imentally, DVP outperforms baselines on aver-
age across 11 downstream datasets.
Notably,
the DVP-PRM integration enables insights into
how individual visual prompts influence classifi-
cation decisions, providing a probabilistic frame-
work for understanding reprogramming.
Our
code is available at https://github.com/
tmlr-group/DecoupledVP
1. Introduction
Model reprogramming (Vinod et al., 2020; Chen, 2024;
Hung et al., 2023) is shown to be an effective approach for
*Equal contribution 1School of Computing and Information Sys-
tems, The University of Melbourne 2School of Computer Science
and Engineering, Southeast University 3Idealism Technology (Bei-
jing). Correspondence to: Feng Liu <fengliu.ml@gmail.com>.
Proceedings of the 4...