Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Authors: Chengyi Cai
Zesheng Ye
Lei Feng
Jianzhong Qi
Feng Liu
Venue: ICML 2025 poster (2025)
Keywords: Model Reprogramming
Visual Reprogramming
Vision-Language Model
Visual Prompting
Primary area: general_machine_learning
TL;DR: Decoupling target task with specialized prompts for each unlock visual prompting's potential.

Abstract: Model reprogramming adapts pretrained models to downstream tasks by modifying only the input and output spaces. *Visual reprogramming* (VR) is one instance for vision tasks that adds a trainable noise pattern (i.e., a visual prompt) to input images to facilitate downstream classification. The existing VR approaches for CLIP train a single visual prompt using all descriptions of different downstream classes. However, the limited learning capacity may result in (1) a failure to capture diverse aspects of the descriptions (e.g., shape, color, and texture), and (2) a possible bias toward less informative attributes that do not help distinguish between classes. In this paper, we introduce a decoupling-and-reweighting framework.  Our *decoupled visual prompts* (DVP) are optimized using descriptions grouped by explicit **c**au**se**s (DVP-cse) or unsupervised **cl**u**s**ters (DVP-cls). Then, we integrate the outputs of these visual prompts with a *probabilistic reweighting matrix* (PRM) that measures their contributions to each downstream class. Theoretically, DVP lowers the empirical risk bound. Experimentally, DVP outperforms baselines on average across 11 downstream datasets. Notably, the DVP-PRM integration enables insights into how individual visual prompts influence classification decisions, providing a probabilistic framework for understanding reprogramming.

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Chengyi Cai Zesheng Ye"
Lei Feng
Jianzhong Qi
Feng Liu
Abstract
(a) Single
(b) Decoupled VPs
Model reprogramming adapts pretrained models to downstream tasks by modifying only the in- put and output spaces. Visual reprogramming (VR) is one instance for vision tasks that adds trainable noise pattern (i.e., visual prompt) to input images to facilitate downstream classi- fication. The existing VR approaches for CLIP train a single visual prompt using all descriptions of different downstream classes. However, the limited learning capacity may result in (1) fail- ure to capture diverse aspects of the descriptions (e.g., shape, color, and texture), and (2) possi- ble bias toward less informative attributes that do not help distinguish between classes. In this pa- per, we introduce a decoupling-and rewiihhgggt framework. Our decoupled visual prompts (DVP) are optimized using descriptions grouped by ex- plicit causes (DVP-cse) unsupervised clusters (DVP-cls). Then, we integrate the outputs of these visual prompts with a probabilistic reweight- ing matrix (PRM) that measures their contribu- tions to each downstream class. Theoretically, DVP lowers the empirical risk bound. Exper- imentally, DVP outperforms baselines aver- age across 11 downstream datasets. Notably, the DVP-PRM integration enables insights into how individual visual prompts influence classifi- cation decisions, providing a probabilistic frame- work for understanding reprogramming. Our code is available at https: //tp:/gggttttttt//// com/ tmlr- prap-ggggpp//ccuueeee
Cause
Cause 2
Chuster
Cluster 2
Cluster *
Descriptions
Cause
Reweight
Reweight
Align
OR
Aligm
Align
Previous Method
By Explicit Causes
By Unsupervised Clusters
Figure I. Difference between (a) existing VR methods that train single VP for all descriptions, and (b) our DVP that trains dlecoupled-nnd-reweighee VPs that are optimized using descrip- tions grouped by explicit causes unsupervised clusters. Learn- able parameters are marked with 'fire's.
adapting models pretrained on abundant data to a specific task by modifying inputs and outputs without altering the model's core architecture or retraining. Among these meth- ods, visual reprogramming (VR) (Cai al., 2024b;a; Tsao et al., 2024; Chen al., 2023), also known adversar- ial reprogramming (Elsayed et al., 2018; Tsai et al., 2020), aims to repurpose pretrained models for downstream image classification tasks by adding trainable noise patterns to the input images. VR for vision-language models (VL.s) (Cai et al., 2025) trains single noise pattern, also known as a visual prompt (VP), to align input images with (i.e., bring the visual features closer to texture features) all descriptions of different downstream classes, shown in Figure (()).
However, attribute descriptions (Cai et al., 2025) are com- plex and diverse, while the learning capacity of a single VP may be limited, potentially insufficient to handle all image-description pairs. This may result the VP success- fully capturing descriptions in certain aspects while failing in others. Additionally, the VP may unreasonably tend optimize less informative descriptions that, while easier to match with images (i.e., having higher cosine similar- ity), exhibit low discriminability between classes. Figure 2 shows an example of a single VP pattern trained on the OxfordFlowers (Nilsback & Zisserman, 2008) dataset for all attributes (Cai et al., 2025), with sample descriptions shown
1. Introduction
Model reprogramming (Vinod et al., 2020; Chen, 2024; Hung et al., 2023) is shown to be an effective approach for
'Equal contribution 'School of Computing and Information Sys- tems, The University of Melbourne "School of Computer Science and Engineering, Southeast University 'Idcalism Technology (Bei- jing). Correspondence to: Feng Liu <fengliu.mi gmail. gmail.com>. com.>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Sample D Descriptions for the A rtichoke
(a) Not Capturing Descriptions of DiverseAApects DiverpteeoooiiiivveAeeeeeeeet
(b) Unreasonable Description Blas
Shape
flower articholr has lampe, rounded, and spily approrance.
Cosine Similarity of Images & Uribules
Cosine Similarity of Images & A Urrbules
Stem/Leaf
The flower actichoke has a bead made up of ovedlapping,
green es.
0.378
0.374
Betore
0,373
0,365
Pedal Color
Artichole petals exhibit. a mix soft purple lavender and green lones.
Reprogramming
Class rtichoke
Ignoored
0,330
0343
Sample D Descriptions for the Globe Thistle
Cosine Similarity of Images & A ttributes
Cosine Similarity of Images & ttributes
Shape
The flower bisslle has large, rouzded, and spiky appearunce.
Stem/Leaf
The flower globe thistle striking flower that grows on tail, spiky stems,
0.374
0.365
0.365
Class Globe Thiste
0:357
Pedal Color
Globe thislle striking blushh- purple petals.
Reppograaminn
0.343
1
03344
Ienored
Figure 2. Drawbacks of training a single VP with all attribute descriptions. Sample descriptions of different aspects for "artichoke' and "globe thistle' are shown the left, while recorded cosine similarities of images and descriptions are the right. Figure (a) illustrates its inability to capture descriptions of diverse aspects, and Figure (b) revcals its unrcasonable bias towards less informative descriptions.
on the left. Three attributes of different aspacts-"shape" "stem/leaf" and 'petal care listed for class 'artichoke' and 'globe thistle". Figure 2(a) shows the cosine similarity between attribute descriptions and the image 'artichoke' be- fore and after reprogramming using the VP pattern. It can be observed that although adding VP enlarges the similar- ity between the input image and the description regarding 'petal color', the description about "stem//caf"-allll being an important aspect for distinguishing artichokes-ho has, con- versely, become more alien. This illustrates the inability of a single VP to capture descriptions of diverse aspects. Figure 2(b) shows the cosine similarity between the reprogrammed image 'artichoke' or "globe thistle' and their corresponding descriptions. Intuitively, both flowers share the same 'shape' (i.c., large, round and spiky), while aspects like 'stem/leaf' and 'petal color' may be more informative for distinguishing between these two classes. However, as 'shape' descriptions have a higher similarity with the images, the VP learning process favors them. It highlights how using a single VP may exhibit a bias towards less informative descriptions.
bility reweighting matrix (PRM) which is optimized during learning through maximum likelihood estimation (MLE), We also show that DVP improves the results by lowering the empirical risk bound.
Section shows the application of DVP to 11 commonly used downstream datasets and four CLIP backbones, demon- strating its effectiveness. The parameter analysis, ablation experiments, and independence tests further validate the rationality of DVP. Remarkably, the visualization results of PRM reflect how individual VPs influence classification decisions, reflecting their different roles and contributions across various downstream classes.
In conclusion, both theoretical analysis and experimental results verify the soundness of DVP. The DVP-PRM integra- tion also offers probabilistic framework for understanding model reprogramming by exploring individual VPs.
2. Related Works
Model Reprogramming. Model reprogramming (Chen, 2024) is a supervised finetuning approach that achieves transfer to downstream tasks while preserving the integrity of pretrained models, by solely modifying the input and output spaces. It has demonstrated promising applications across pretrained vision (Chen et al., 2023; Tsai et al., 2020; Cai et al., 2024a; Jín et al., 2025), graph (Jing et al., 2023), acoustic (Yang al., 2021: 2023; Hung et al., 2023; Yen et al., 2023), and language models (Hambardzumyan et al., 2021; Vinod et al., 2020; Jin et al., 2024). Comparatively, it not only ensures the completeness of the pretrained model, preventing catastrophic forgetting (Kirkpatrick et al., 2017) but also enables architecture-agnen stream tasks with minimal parameter adjustments. The robustness of model reprogramming has recently been in-
In this paper, we first formulate the problem and then in- learning framework
troduce a (as shown in Figure i(b)) to avoid the above drawbacks of a single VP. Under our framework, decoupled VPs will be optimized for different description partitions, and the impor- tance of descriptions distinguished through reweighting (see Section for details).
In Section 4, for the decoupling step, propose decou- pled visual prompts (DVP) that are optimized using descrip- tions grouped by explicit causes (DVP-cse) or unsupervised clusters (DVP-cls). Explicit causes can be determined by querying a large language model (LLM) while unsupervised clusters can be resolved with K-means. For the reweighting step, the output of DVP will be integrated using a proba-
transfer to down-
2

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
vestigated (Chen et al., 2025; Zhou al., 2025).
When CLIP is used for downstream classification task de- fined over X x xyT,, where X C TR and y" respectively represent the dr-dimensional image space and the label space, we often use textual descriptions A S V to represent class labels as texts that CLIP can process. Often, a descrip- tion can be implemented as a template with placeholders, e.g., "This is a photo of [Class Name]." (Radford et al., 2021) or a set of attributes for the class, c.g., "[Class Name] is [Attributes]." (Cai et al., 2025). Let A(y") be the subset of m descriptions for a class y² E )V', with description sets for different classes being disjoint, A(v," ( (AA()))) = Ø for y/ + w3 and A - UUTTET A(v").
Input VR (Cai et al., 2024b;a; Chen et al., 2023) is type of model reprogramming, primarily used to apply pretrained models to downstream image classification tasks by adding trainable noise patterns to the input images. The main strate- gies include padding trainable parameters around the images (Chen al., 2023; Tsai et al., 2020: Tsao et al., 2024) or adding watermarks (Bahng et al., 2022: Oh et al., 2023) to rescaled images, which has been successfully applied to both unimodal image classifiers (Chen et al., 2023; Cai et al., 2024a) and VLMs (Oh et al., 2023; Zhang ct al., 2024).
Prompt Learning. Slightly different from model repro- gramming, prompts (Jia et al., 2022) serve as learnable weights that can be attached to any position of pretrained model. Therefore, prompt design often needs to consider the specific architecture of the pretrained model. Prompts can be added as text prompts (Zhou et al., 2022b;a) among input words, as VPs (Chen et al., 2023; Oh et al., 2023; Tsao et al., 2024) overlaying input images, as token prompts (Wang et al., 2023) within self-attention layers, or as map- pings (Khattak et al., 2023) between modalities to connect the text and image embeddings.
Let flogits XT x A - RD'" be the logit vector over the label space y" If an input image x= E X matches CLIP's input dimension (dT = ds), the logit for class y" is
[fogita T A))l, A( (felp(a", ,a));
where a is a description and agg(.) is an aggregation func- tion, typically max(-) or avg(.).
VR on CLIP. For mismatched downstream input shapes, i.c. dr # dg, standard input VR (Cai et al., 2024b) uses an input transform fin RdT - Ieds defined as fin (x & pad(x")) + 6, where pad(.) function pads zeros around the images and VR patterns 5 E IRds is a trainable visual prompt. The logit for a class y² E y given x:' and prompt ó is:
Specifically, adding VPs to input images for reusing a pre- trained VLM to downstream classification tasks is equiv- alent to VR methods for VLMs. Among recent research, VP (Bahng et al., 2022) overlays watermarking patterns on images, AR (Tsai et al., 2020; Chen et al., 2023) pads patterns around images, BIackVIP (Oh et al., 2023) aims at black-box transfer learning, DAM (Huang al., 2023) partitions the image set for divide-and-conguce training, and AttrVR (Cai et al., 2025) introduces attributes to better align images with texts. These existing methods optimize single VP patterns for all descriptions. Our proposed DVP strives to train VPs with diverse roles and contributions, enhancing both learning capability and interpretability.
Sogina (n (xF:;, (6. y²
= ag8 (f (1)
Then, the normalized probability will be obtained by
exp([Tiian (a (5.A))))). Per Po = Exeyr ( (w (6.A))
(2)
The parameter ó in VR can be learned by minimizing the negative log-likelihood, computed over a downstream train- ing set D = {xx]] 2/ y
iid XT yJT as,
3. Preliminaries and Insights
= arg ming - ΣSn (u (vFFe1;:
(3)
We follow standard protocols for VR on CLIP (Chen et al., 2023; Cai et al., 2025). See Appendix D.1 for a setup com- parison between our problem and text prompt tuning (Zhou et al., 2022a), and Appendix D.2 for a summary of notation.
where N n x | is the size of training dataset with samples per class.
Representing in the Matrix Form. To rewrite Eq. (1) in the matrix form, for certain input image x'', we can de- fine a row vector Ma = [ā1, ......, ا|داة being the CLIP output feip (f (a (()); across different attribute descrip- tions and another row vector M,- = [11. ....... 7prT be- ing the logits fyr (=";; A) across different downstream labels. Then they can be related by a reweighting matrix w E R|A[X that Ma w = My:
CLIP-based Image Classification. CLIP (Radford al., 2021) is pre-trained VLM with an image encoder fing 8 + Z that projects input images from -ds-dimensional image space X C dsds to embedding space 2, and text encoder Sixt : V + Z that projects texts V E V to the same embedding space Z. The similarity between an image S E 2 and a text V E is:
...
ā1
W1,1
WLIT
01
feup(15. V) = cos (fmgg(²))
(rxcs (V)) /T.
52
(4)
where cos(.:) is cosine similarity and r is the temperature.
[[Ai
WAALDT
3

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
(a) Before Training
(b) Training Iterations
Method 1:
Method 2:
y
=
Probability Reweighting
Stop2
Cause (cse)
Cluster (cls)
Downsbree Task
Decoupted Visual
Prompts (DVP)
Traaean Encoda
Matrix (PRM) w
GPT
ttributes
W1,1
W1e e
W2,1
W2,
Ground Ground-truuhh truth Labels
Cause 2
Carse t
CLIP-Text Encoder
Logits
Cause
*
6,
mc
CLIP-Text Encoder
V
K-means
Method 1: Cause (cse)
Method 2: Cluster (cls)
X
mc
Wmc.1
Wmeee
Step1
A = UA,
A. "ay...,,A,, an LLM based different causes (DVP-cse) by clustering existing attribute descriptions (DVP-cls). During training, VPs are trained for their corresponding description partitions separately, after being integrated and reweighted by PRM w. In each iteration, w is first estimated, followed by the update of VPs A ; 5.), w.r.t. negative log- sigllikeiioodd (i.e., cross-entropy loss in classification tasks).
Figure 3. Pipeline of DVP. Before training. the description partitions A.,...,
are determined either by generating descriptions using
where each element Wp.g in w represents the contribution from äp (i.e., CLIP output of the p-th attribute descrip- tion) to Do (i.e., logit output for the q-th label). Here, Eq. (4) implies that agg(-) implements a fixed reweight- ing scheme. Concretely, if max(-) is used in Eq. (I), then E {0, 111AIx12" 4p.q is set to be only when ap is the maximum CLIP output for the g-th class's attribute description, otherwise Wip.g = 0. If avg(-) is used, then w E {0, 1 | and Wp.e = 1/m only when the p-th mi attribute a description related to the q-th class.
4. Decoupling Visual Prompts
This section details our DVP method, which implements the decoupling-and--eweighhn
strategy outlined in Section 3. Figure illustrates the overall DVP pipeline. We begin by describing two ways for partitioning class descriptions (Section 4.1), a preparatory step before training. We then explain how our Probabilistic Reweighting Matrix (PRM) is estimated (Section 4.2). We use iterative training, where the VPs set A and the PRM w are jointly optimized (see Section 4.3). We lastly show DVP can theoretically attain a lower empirical risk than standard input VR (Section 4.4).
Then can replace agg(-) with fixed ax and rewrite the logits of label B4 in Eq. (1) as Dogiis (xT;; wfx, A)),, (mnpp()(==)))) up), where "p.y
Ea,ed fix "p. entry in fix corresponding the p-th description up and
fixxx denotes the
4.1. Partitioning Descriptions
A core idea of DVP is to train distinct VPs {ō), (6,))..,,.).. each specialized for a corresponding partition of the full de- scription set f. A.). We next present two strategies for partitioning descriptions, namely DVP-cse and DVP-cls.
the q-th class V4- wix is non-zero only for ap that con- tribute to the aggregation for the class Vy. Eq. (2) under this generalized logit definition now becomes:
By Semantic Causes (DVP-cse). Following Cai al. (2025), who used LLMs to generate attribute description sets for each class, we leverage LLMs, e.g., GPT series (Brown et al., 2020), to create semantically coherent description partitions for each cause (i.c., distinguishing aspects for the classification task). First, we query an LLM to identify v primary "causes" (see Appendix A.3). For each i-th cause, we formulate specific query, cause .pommpt, = "Describe the [i-th Cause] of the [Task Info.] image [Class Name]", which is fed into an LLM to generate a set of descriptions A, for each class y² The i-th global partition is then A = U,yee" A((yT), ). For computational efficiency, these descriptions are pre-converted to text embeddings:
Pvr 11 ;8, A) exp([[iinns Sa A)) AT) Every expl Tranee (at Saall A) A
(5)
Framework. Consid- ering Eq. (4), a limited M, in current VR methods may not capture descriptions of diverse aspects from A, shown in Figure 2(a), and trivial may cause the unreasonable description bias in Figure 2(b).
A Decoupling-and-Reweiee
Therefore, a Figure 1(b) can be proposed. Firstly, to enhance learning capacity, a decoupled VP set A = (6I.....6,, of size v can be optimized separately with partitions A. of A to get a better Ma Besides, a more precise Probabilistic Reweighting Matrix W PRM E [o, 1||d|x with continu- ous values can be estimated for reweighting the outputs for different attribute descriptions.
aadpding--nn-rewegghiiiii framework like
& (fxa(a) Ac). for i =,...,,
(6)
Each embedding set ε, (representing Ad) guides the learning of its dedicated prompt 8,
4

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
By Unsupervised Clusters (DVP-cls). Alternatively, when using L.Ms to generate descriptions is restricted, we can still partition existing set of descriptions A. We first embed all descriptions into & = (a) | a E A}. We then use K-means algorithm (MacQueen et al., 1967) to partition these embeddings into v clusters:
Algorithm 1 Pipeline of DVP
I: Input: Few-shot training data D {(0} vj 1= description set A, hyper-parameters k. v, epoch number E, and pre-trained CLIP model selip with fix and Sing encoders
Output: DVP set A A= (ō1....,, 5,) and PRM w # Before training: divide descriptions
[E ) KMeans(E, u).
(7)
4: Obtain & = U & by causes (method DVP-cse) us- ing Eq. (6) clusters (method DVP-cls) using Eq. (7)
This way each cluster & implicitly refers to description partition A; = falfaa(a) E }, which is then used to train its respective prompt 8,
5: # Begin training
6: Initialize 6; + 0 for i = I,...,
7: for - 1 to E do
Compute/update CLIP output (Ma in Eq. (4)
4.2. Reweighting Descriptions
Compute felip (Sin [x]] (e]])))))) a) for j I...., N. a.a
Probabilistic Reweighting Matrix (PRM). As discussed in Section 3, PRM accounts for quantifying the relationship between description similarity and class logits. Let ap be the p-the description and Hy be the q-th class. We estimate the conditional probability pVV = ap vg), denoted by wprm Each entry wp.g reflects the contribution of de- scription ap class Vg- If ap is not designated description for class My (i.e., Op E A(yg)) then wprm = 0 (Proposition B.I, Appendix B). Otherwise, wprn for Og E A(y,) can be estimated via MLE from counts on the training set D.
A using &
10:
# Step 1: update reweighting matrix (w in Eq. (4))
11:
+ wPrm using Eq. (8) and Eq. (9)
12:
# Compute/update logits output (Mg in Eq. (4))
13:
Compute Dg <- EayeA Wp.gSelip (Jin (of (p))))
q = -=,.... | for j I....,N N
14:
# Step 2: update DVP patterns
15:
d; x 68 using Eq. (10) for i = I....,U
16: end for
prm P.. = P(V = apl appx==== uy V E eA(g)))
description partition A.. Specifically, for each d. the objec- tive mirrors standard VR's objective (Eq. (3)) over D:
Narlan v4)
(8)
E Ea'eA( NDla'.u,)
- N Σ Pog 8i, wom Ai).
Here, ND (a, y) is the co-occurrence count of description a with class y (see Proposition B.2, Appendix B for details).
ming,
(10)
Counting for Few-shot Settings. To handle data sparsity in few-shot settings (where (ap 124) might be sparse, mak- ing MLE less effective), we smooth these counts. Specifi- cally, for each training sample (x:f. v,y',, instead of checking for an exact match with ap. we find the is descriptions in A most similar to the reprogrammed image Sin (x))))) Note that only the specific d; E A is used if a description belongs to Ag. Let this set be K(x]] k). The count is then:
Here, pvr derives from partition-specifie logits (using A,, 6, and wPrm), contrasting a single-promp aggregation (with A, 8, and fxxx as in Eq, (5). This process is applied inde- pendently to all E A. See Appendix B.2 for details.
4.4. Justification: Empirical Risk Reduction
We analyze DVP by showing its potential to achieve a lower empirical risk on the training data than standard VR.
ND 82) = En 12y 1a) 1aap E K(cJ,A)), (9)
Definition 4.1 (Empirical Risk). Consider an input image space X, a discrete label space y, a distribution D' defined over X x y. Let D = {(=] v/ )) be a training set drawn i.dd from D'. For a classifier f f X + y parameterized by θ the empirical risk (e) is defined as the average negative log-likelihood as
where 1(-} is the indicator function and k is a hyper- parameter. That is, substitute Eq. (9) into Eq. (8) to obtain a more robust estimation wrm in practice.
4.3. Iterative Training
R = - N log logp(u;1z;;0).
Once the partitioning & and A are obtained through either DVP-cse or DVP-cls, the parameters A and wPem are opti- mized through an alternating optimization procedure (see Algorithm 1 for details). In each training epoch, we first update prm using Eq. (8) given the current A. Then, we update each & A independently using only its assigned
Empirical Risk of Standard VR. According to Definition 4.1, for standard VR, the parameters are @ = {5), and the reweighting matrix ffx is pre-determined (c.g., by max(-) or avg(-) aggregation). Thus, the empirical risk of standard
5

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Table 1. Accuracy comparison of different methods trained on 16-shot downstream classification tasks, using ViT-Bl6- CLIP as the pretrained model (Mean % + Std %, ours highlighted and the highest is in bold). See Appendix C.7 for parameter numbers.
METHOD
AIRCRAFT
CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
AVG.
VP
32. 1.0.6
93.5.0.1
65.5.03
61 5
91.2.0.3
82.5+0.4
82.3-o.1
91. 91.0+0.3
65.8.0.2
73.8
79. 19.3
74.4
AR
31.7503
95.5
68. 38.0
62.0.0.1
93.4
85.9
85.2.0.1
92.7.0.1
67 940.
18.1
81.6
76.5
ATTRVR
36.620.3
95.7a0.1
68. 3,0.3
65 620.
93. 330.3
92.9x0.4
85.9
93.3s0.0
69.6sn.1
79.0an.6
82.6s0.4
78.5
DVP-CSE
40.3=0.1
96.2-0.1
72.5=0.2
66.7.0.4
93.920.1
95.4.0.1
85.6
13.1
71.
81.7,0.3
84.6
80.1
DVP-CLS
38.7
96.0+o.o
65.5.0.7 70.8
94.1+0.3
95.0-0.2
85.7-o.0
93.3+0.1
71.1
82.0
84.4
79.7
VR methods is
Lemma 4.5. For any fixed DVP parameterization A, let worm be the optimal PRM that minimizes Rep "Ap (A, wPm)). Then ( (A, 2 Re (A, wpr"") holds for any wix
(8, = N
Σ log Pog (v] loo 8, wix A),
Lemma 4.5 (proved in Appendix B.4) indicates the effec- tiveness of reweighting the contribution from descriptions to labels. PRM results in a lower empirical risk compared with the pre-defined and fixed weights.
where pd is derived from logits in Eq. (1) using fix A is the complete set of descriptions.
Empirical Risk of DVP. For DVP, whose parameters are @dvp = {A, wymm ). Given the risk (Eqq. (10)) jointly ap- plied to all 6, A, The empirical risk of DVP is then
Corollary 4.6. Let 8 be the optimal VP for standard VR with wix Let {A", _.prm" } be the optimal parameteriza- tion for DVP that minimizes Revve (A, W.r)). Then, it holds that feg (8+ w 2 iddvp A",, ).
N Σ
ReMP wPm) =
Σ log
[a]: 6,, "e,
Remark 4.2. Reducing all 8, A to the same ô and substi- tuting prm with fix reverts DVP to standard VR.
Corollary 4.6 (proved in Append B.5) indicates that the DVP framework, by virtue of its increased capacity through decoupled prompts and adaptive reweighting, has the po- tential to achieve lower or at least equal empirical risk on the training data compared to standard VR, effectively preventing underfitting in downstream tasks that may exist when applying VR methods.
Given the likelihood- form of empirical risk (Defini- tion 4.1) used in our context, we further define optimally achievable empirical risk to compare the theoretical capa- bility of standard VR and DVP.
Definition 4.3 (Optimally Achievable Empirical Risk). Con- sider the training set DD and the classifier f parameterized by θ as with Definition 1. The optimally achievable em- pirical risk over θ is defined as
5. Experiments
Baselines and Benchmarks. To evaluate DVP, we follow Cai et al. (2025) to conduct training and testing on pretrained CLIP models (with various image encoder architectures) for 16-shot downstream classification tasks. Averaged accu- racies of three seeds are used for evaluation. These tasks cover various classes and domains, including patterns, ac- tions, scenes, etc, with more information in Appendix A.1.
(0) infe (e) infe (- Ex- 1ogp(yy 6)).
where the infimum is taken over all possible parameteriza- tions θ of f, and, if the infimum is attainable, we denote
O" = arg ming (+) as the optimal parameterizatiin uniquely achieving R²)
The following three VR baselines are included: (1) VP (Bahng et al., 2022), a VR method in which learnable pa- rameters are overlaid on rescaled downstream images; (2) AR (Tsai et al., 2020; Chen et al., 2023), a VR method that pads learnable parameters around the image; and (3) Att (Cai al., 2025), a method that guides the learning of VP patterns through attribute descriptions all baseline methods learn single and non-decoupled VPs. Both DVP-cse and DVP-cls are tested, with the hyper- parameter à chosen to be 3 (see Table 7). The number of VPs in DVP-cse is set to be v = 3, while that in DVP-cls is a
We now establish two lemmas and corollary.
Lemma 4.4. Let 8* be the optimal VP for standard VR, min- imizing Reg (xx for a fixed reweighting matrix w fix Let Ar be the optimal DVPs that minimizes Rejn (A, (xx) i.e. DVP using the same fixed fix standard VR. Then, we have 18 fxx) 2 fe ",ee, w
Lemma 4.4 (proved in Appendix B.3) demonstrates the ad- vantage of decoupling VPs and training them separately for different description partitions. The decoupling step contributes to the reduction in the optimally achievable em- pirical risk by improving the learning capacity of the VPs.
We note that A is different from uix in Lemma 4.4.
6

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Table Ablation studies of DVP-cse and DVP-cls, using ViT-B16-based CLIP as the pretrained model (Mean % Std %, ours are highlighted and the highest in bold).
METHOD
AIRCRAFT
CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
AVG.
DVP-CSE
40.3s0a
96.2.0.
72.5.02
66.7.0.
93.9
95.4.0.
85. 15.6...1
93.1
71.1.0.2
81.740.3
84.6
80.1
W/O CSE
36.4
95.8s0
69. 1s0.1
65.3
94.1+0.
93.6,022
85.7.0.1
93.1
70.0.0.1
20.2
82, 02.8
78.7
W/ VR
27.5
93.2s0.
63.920.4
57.020.3
45.920.
33.620.2
83.920.0
91.3a0.1
62.3sn.2
38.3
60.3an.4
67.0
W/O @Pm
38.0a0.1
95.8.0.
70.4s0.4
67.7x0.3
94.8
25.0
85,8
93.5x0.1
67.1.n.1
80.3x0.5
84.9
79.4
DVP-CLS
38.7.0.
96.0-a.
70.8.0.
65.5
14.1
95.0.0.2
85.7+0.a
93.3
71.1.0.2
82.0.o.1
84.4.0.0
79.7
w/O CLS
36.4
95.8e0.1
1
65.3
14.3
93.6.0.2
85.7.0.1
13.1
70.0
80.2
82.
78.7
W/o VR
26.0s0.0
93.9s0.1
13.1
55.0x0.2
52.2.0.4
13.
84.8s0.0
91.3an.1
64.4s0.1
70.0an.4
60.8so.
67.7
W/O uPm
37.8
96.0.0.2
70.6
65.4x0.7
93.9t0.2
94.2s0.3
85.8s0.1
93.3an.1
69.4s0.1
80.7
84.6.0.1
79.2
Downstream Task: Flowers
Downstream Task: Texture
Downstream Task: Aircraft
An=VR
DVP-
DVP-ds
70.0
AtrVR
DVP--00
DVP-cl
42.5
AnrVR
DVP-c5e
DVP-cl
Accuracy (%)
96.0
Accuracy (%)
67.5
40.0
Accuracy
94.0
65.0
37,5
92.0
62.5
35.0
60.0
2
3
2
3
Number VR Pattems
Number of VR Patterns
Number of VR Patterns
Figure 4. Accuracy comparison of different VR methods with the number of VR patterns (i.c., VPs) o E (1,2,3,5,7) 7}. Pre-trained ViT-B16-based CLIP is used. The striped area indicates the error bars.
hyper-parameter that represents unsupervised clusters and is chosen as shown in Table 8. More information about hyper- parameters in Appendix C.I. Implementation details are in Appendix A.2 and generated causes are in Appendix A.3.
DVP shows no improvement on the Food dataset. This might be attributed to the primary distinguishing factors for food being smell and taste, while visual information is less useful. As result, decoupling VPs for better visual learning will not lead to classification improvements.
Performance Comparison. The results on the ViT-16- based CLIP model are shown in Table 1. It can be observed that both DVP-cse and DVP-cls demonstrate strong perfor- mance. DVP-cse outperforms the baseline method Attr by an average of 1.6% across 11 datasets, while DVP-cls achieves an average improvement of 1. 1.2%. Except for the Food dataset, applying DVP-ese or DVP-cls achieves accu- racy equal to or higher than Att on all other datasets.
Ablation Studies. The ablation studies for DVP-cse and DVP-cls are presented in Table 2. "W/o cse" and "w/o cls" refer to excluding decoupled VPs, reducing both methods to training a single VP. "W/o VR" indicates the zero-shot learning results without training the VR pattern, while "w/o Drmm" represents results without PRM.
Without decoupling, a single VP struggles to capture di- verse descriptions, leading to unsatisfactory performance, especially on complex aineggrined classification tasks such as Aircraft, Cars, and Flowers. Similarly, omitting the op- timization of VP results in poor accuracy on downstream tasks with significant differences from the pre-training do- main, such as remote sensing datasets (e.g., EuroSAT, Re- sisc). When the reweighting matrix prm is not applied, the performance is notably affected in datasets with large number of categories (e.g., SUN with 397 classes) or com- plex descriptions (e.g., UCF for action classification). In summary, every component of DVP indispensable for achieving optimal performance improvements.
DVP-cse is designed to train VPs for descriptions grouped by different causes. Consequently, it achieves better results when multiple specific aspects are crucial for classification. For instance, the Cars, Aircraft, and Flowers datasets contain fine-grained classes that vary along multiple cause factors (i.e.. color, shape), allowing DVP-cse to achieve improve- ments of 4.2%, 3.7% %, and 2.5% over Attr respectively.
Meanwhile, DVP-cIs is a method that learns VPs based on descriptions after unsupervised clustering. It tends to group classes with similar features into a cluster for one single VP. For datasets like UCF (i.e., different actions), or fine- grained datasets like Cars and Aircraft, classes can be clearly divided into subcategories based on similarity. Therefore, DlP-cls achieves better performance than Att by 3%, 2.5%, and 2.1 %% on these datasets, respectively.
Impact of the Number of VPs. Figure 4 illustrates the results of VR methods with various numbers of VPs. Sim- ply increasing the number of VPs without employing the
7

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Weight. Proportions of Different Causes after Training VR Pattems for Some Classes (Downstream Task; Flowers; Method: DVP-cse)
Globe Thistle
Silverbush
Grape Hyacinth
Corn Poppy
12.5%
16.7% 12.5%
18.8%
12.5%
35.4%
43.8%
12.5%
68.8%
75.0%
70.8%
20.8%
flower structure
shape of the petals
color
Figure Visualization results of DVP-cse, showing weights causes for classifying certain classes, using ViT-B16-basee CLIP.
Top Classes within Different Clusters (Downstream Task: Flowers; Method: DV PVPls)s)
Cluster One
Cluster Two
Cluster Three
sword lily
10.97
barbeton daisy
10.97
rose
10.68
canna lily
11.04
oxeye daisy
11.63
hibiscus
10.85
fire lily
11.79
sunflower
11.64
azalca
11.15
Sorted by Distance the Cluster Center
Sorted by Distance the Cluster Center
Sorted by Distance the Cluster Center
Figure 6. Visualization results of DVP-cls, showing the top classes closest to the description cluster centers, using HTT-B16-based CLIP.
framework proposed in this paper may even lead to a decrease in accuracy due to po- tential overfitting. However, when the DVP approach is used-whether in the form of DVP-ese DVP-cls-the accu- racy steadily improves. This suggests that the performance improvement achieved by DVP is not merely due to the increase in parameters, but is attributable to the benefits of our learning framework.
Table 3. Average accuracy of different VR methods 11 datasets, using different backbones as CLIP visual encoders (Mean Accu- racy %, ours are highlighted and the highest is in bold, RN stands for ResNet).
VP
AR
ATTRVR
DVP-CSE
DVP-CLS
RN50
53.5
60.4
64.6
66.6
RN101
57.5
62.7
67.2
69.6
68.8
VIT-B
68.3
66.3
69.8
71.3
71.0
VIT-B16
74.4
76.5
78.5
80.1
79.7
Visualization of Cause Weights (DVP-cse). DVP- cse, since each trained VP corresponds specific cause, for single class, the weights corresponding to individual VPs in the reweight matrix can be summed to calculate the contribution of that cause to the predicted label. Figure 5 illustrates the contributions of three causes for four classes in Flowers dataset. Specifically, the spherical spiny shape of the globe thistle, the trumpet-shaped petals of silverbush, and the bluish-purple petals of grape hyacinth make these three flowers distinct, matching the statistical results flower structure, petal shape, and color are the primary causes for classifying these flowers, respectively. In contrast, corn poppy is relatively ordinary in various aspects, resulting in similar contributions across different reasons. can be observed that the reprogramming process determines the priority of causes driving the final class predictions. More results Texture and Aireraft datasets are detailed in Appendix C.2.
Visualization of Cluster Components (DVP-cls). Com- pared with the VPs in DVP-cse, which have clear corre- spondences to specific causes, the VPs in DVP-cls tend to correspond descriptions of classes with commonalities. Figure 6 shows the classes with the closest average distance to the description cluster centers corresponding to different VPs. It can be observed that different types of lilies are grouped into one cluster, while daisies and sunflowers with similar shapes are grouped together, and flowers with similar colors are also grouped together. This explains the results in Table 1, where DVP-cse is more suitable for classification tasks with multiple contributing causes, while DVP-cls is better suited for classification tasks with distinct subclass hierarchies. More results are detailed in Appendix C.3.
Results on Different Backbones. Since VR methods modify only the input images, they are compatible with any
8

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
which decouples VPs without introducing additional parameters, confirming that our decoupling-ande reweighting strategy improves performance even when no extra parameters are used. In Appendix C.8, we demonstrate the applicability of our DVP-PRM framework to different VR methods. In Appendix C.9, we lastly present results showing how performance varies with different amounts of available training data.
2
3
4
4-
5
5
5
DVPP-sse
0
DVP--cls
Figure Pairwise HSIC between image embeddings apply- ing non-decoupled (i.e., Att) decoupled VPs (i.e., our DVP). Experiments are done on the Texture task with ViT-16-based CLIP.
6. Conclusion
In this paper, to better align downstream images with de- scriptions for classification, we proposed a decoupling-ande reweighting framework that learns DVP based on specific causes or unsupervised clusters, combined with a PRM to reweight the descriptions. DVP is shown to be effective both theoretically and empirically. By exploring and analyzing the distinct roles and contributions of individual VPs, we offer new insights into model reprogramming.
model architecture. Table 3 presents the classification accu- racy averaged across all datasets when using different image encoders (i.e., ResNet or ViT) in CLIP (detailed results are in Appendix C.4). It can be observed that both DVP- cse and DVP-cls achieve improvements over all baseline methods, regardless of whether smaller (RN50) or larger (ViT-B16) pre-trained models used. The performance improvement of DVP becomes more pronounced for smaller image encoders, indicating that DVP can compensate for the limitations of pretrained models.
Acknowledgement
CYC, ZSY, and FL are supported by the Australian Research Council (ARC) with grant number DE240101089, and FL. is also supported by ARC with grant number LP240100101, DP230101540 and the NSF&CSIRO Responsible Al pro- gram with grant number 2303037. Jianzhong Qi is supported in part by the ARC via Discovery Project DP240101006 and Future Fellowship FT240100170. This research is also supported by The University of Melbourne's Research Computing Services and the Petascale Campus Initiative. We sincerely appreciate the time and dedication of the reviewers in carefully reviewing our manuseript.
Discussion about Independence. We further investigate the independence of different VPs using the Saabbert-cchiidl In- dependence Criterion (HSIC) (Gretton et al., 2007). HSIC is non-parametrie test where smaller values indicate greater statistical independence between two random variables. We calculate pairwise HSIC values (with 3) between the embedding distributions of images after adding different VPs. This done for a baseline approach (i.c., AttrVR, which does not decouple VPs) and the proposed decoupling- based approaches (i.e., DVP implemented under both DVP-cse and DVP-cls). Experiments
conducted on the Texture task, which features relatively simple images to highlight the differences. Results in Figure 7 show that embeddings obtained by decoupled VPs exhibit lower sta- tistical dependence between different VPs, which might be a factor that contributes to the improved performance.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
We conduct several additional periments to further validate our approach and explore its characteristics. In Appendix C.5, we analyze the time cost of various VR methods and conclude that DVP can achieve performance improvements under the same time cost con- straints. Appendix C.6, we analyze different causes and investigate what makes cause effective for VR us- ing DVP-cse. We conclude that the most suitable causes should (1) be related to visual information, and (2) have causal relationship with accurate classification. This experi- ment demonstrates the application of DVP-cse in evaluating the importance of various causes for VR in classification tasks. In Appendix C.7, we discuss the potential param- eter overhead associated with DVP, and introduce a vari-
More Experiments.
References
Bahng, H., Jahanian, A., Sankaramarayanan, S., and Isola, P. Exploring visual prompts for adapting large-scale models. arXiv, 2022.
Bossard, L., Guillaumin, M., and Van Gool, L. Food- 101- mining discriminative components with random forests. In ECCV, 2014.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. NeurIPS, 2020.

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Cai, C., Ye, Z., Feng, L., Qi, J., and Liu, F. Bayesian-guided label mapping for visual reprogramming. In NeurIPS, 2024a.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 2019.
Cai, C., Ye, Z.,, Feng, L.,, Qi J., and Liu, Sample-specific masks for visual reprogramming -basd prompting. In ICML, 2024b.
Huang. Q.. Dong, X., Chen, D., Zhang, W. Wang, F., Hua, G., and Yu, N. Diversity-aware meta visual prompting. In CVPR, 2023.
Cai, C., Ye, Z., Feng, L.., Qi, J., and Liu, F. Attributebbased visual reprogramming for image classification with clip. In ICLR, 2025.
Hung, Y.N., Yang. C.-H. H., Chen, P-Y.,, and Lerch, A. Low-resource music genre classification with cross- modal neural model reprogramming. In ICASSP, 2023.
Chen, A., Yao, Y., Chen, P.-Y., Zhang, Y., and Liu, S. Un- derstanding and improving visual prompting: A label- mapping perspective. In CVPR, 2023.
Jia, M., Tang, L.,, Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In ECCV, 2022.
Chen, P.-Y. Model reprogramming: Resource-efficient cross-domain machine learning. In AAAI, 2024.
Jin, C.,, Li, Y., Zhao, M., Zhao, S., Wang, Z., He, X., Han, L., Che, and Metaxas, D. N. Lor-VP: Low-rank visual prompting for efficient vision model adaptation. In ICLR, 2025.
Chen, Y. Shao. S., Huang, E., Li, Y., Chen, P.-Y., Qin, Z., and Ren, K. Refine: oversion-free backdoor defense via model reprogramming. In ICLR, 2025.
Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F,, Pan, S., and Wen, Q Time-LLM: Time series forecasting by reprogramming large language models. In ICLR, 2024.
Cheng. G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 2017.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In CVPR, 2014.
Jing, Y., Yuan, C., Ju, L., Yang, Y., Wang, X., and Tao, D. Deep graph reprogramming. In CVPR, 2023.
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. Maple: Multi-modal prompt learning. In CVPR, 2023.
Elsayed, G. F., Goodfellow, I,, and Sohl-Dickstein, J. Ad- versarial reprogramming of neural networks. In ICLR, 2018.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Trabska-Barwinte A., et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
Fei-Fei, L.. Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In CVPR workshop, 2004.
Kolmogorov, A. Foundations of the theory of probability. 1956.
Fisher, R. A. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathemati- cal or Physical Character, 1922.
Krause, J., Stark, M., Deng. J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In ICCV workshops, 2013.
Gretton, A., Fukumizu, K., Teo, C., Song, L., Schölkopf, B., and Smola, A. A kernel statistical test of independence. NeurlPS, 2007.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2016.
MacQueen, J. et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium Mathematical Statistics and Probability, 1967.
Hambardzumyan, K., Khachatrian, H., and May. J. Warp: Word-level adversarial reprogramming. In ACL-IJCNLP. 2021.
Harold, J., Kushner, G., and Yin, G. Stochastic approxima- tion and recursive algorithm and applications. Application of Mathematics, 1997.
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine- grained visual classification of aircraft. arXiv, 2013.
10

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Nilsback, M.-E. and Zisserman, A. Automated flower classi- fication over large number of classes. In Indian Confer- ence Computer Vision, Graphics & Image Processing, 2008.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In CVPR, 2022a.
Zhou, K., Yang. J., Loy, C. C., and Liu, Learning to prompt for vision-language models. IJCV, 2022b.
Oh, C., Hwang, H., Lee, H.-y., Lim, Y., Jung, G., Jung, J.. Choi, H., and Song, K. Blackvip: Black-box visual prompting for robust transfer learning. In CVPR, 2023.
Zhou, S., Cheng, X., Xu, H., Yan, M., Xiang, T., Liu, F, and Feng. L. Endowing visual reprogramming with ad- versarial robustness. ICLR, 2025.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In CVPR, 2012.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Soomro, K., Zamir, R., and Shah, A dataset of 101 human action classes from videos the wild. Center for Research in Computer Vision, 2012.
Tsai, Y.-Y., Chen, P.-.,, and Ho, T.-Y. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In ICML, 2020.
Tsao, H.-A., Hsiung, I.., Chen, P.-Y., Liu, S., and Ho, T.-Y. Autovp: An automated visual prompting framework and benchmark. In ICLR, 2024.
Vinod, R., Chen, P.-Y., and Das, P. Reprogramming lan- guage models for molecular representation learning. In NeurlPS, 2020.
Wang, W., Sun, Y., Li, W., and Yang, Y. Transhp: Image classification with hierarchical prompting. In NeurIPS, 2023.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Yang. C.-H. H., Tsai, Y.-Y., and Chen, P.-.. Voice2series: Reprogramming acoustic models for time series classifi- cation. In ICML, 2021.
Yang, C.-H. H., Li, B., Zhang, Y., Chen, N., Prabhavalkar, R., Sainath, T. N., and Strohman, T. From english to more languages: Parameter-efficient model reprogramming for cross-lingual speech recognition. ICASSP, 2023.
Yen, H., Ku, P.-J., Yang, C.-H. H., Hu, H., Siniscalchi, S. M., Chen, P.-., and Tsao, Y. Neural model reprogramming with similarity based mapping for low-resource spoken command classification. In INTERSPEECH, 2023.
Zhang, Y., Dong, Y., Zhang, S., Min, T., Su, H., and Zhu, Exploring the transferability of visual prompting for multimodal large language models, CVPR, 2024.

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
A. Appendix 1: More Training Information
A.1. Dataset Information
Table 4. Dataset Information
AIRCRAFT
CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
INFO.
TASK
aireraft
model
object
fine-grained automobile
texture
remote sensing land cover
flower
food
pet
scene
uction
sensing scene
remote
CLASS
100
100
196
47
10
102
101
37
397
101
45
NUM.
BATCH
64
64
64
64
64
64
64
64
64
64
64
SIZE
We follow the prior work (Cai et al., 2025) set up our benchmark, employing the same methodology split the 16-shot training, validation, and test sets. All datasets are publicly available and listed follows: FGVCAireraft (Aircraft) (Maji et al., 2013), Caltech101 (Caltech) (Fei-Fei et al., 2004), StanfordCars (Cars) (Krause et al., 2013), Texture (DTD) (Cimpoi et al., 2014), EuroSAT (ESAT) (Helber et al., 2019), Flowers102 (Flowers) (Nilsback & Zisserman, 2008), Food101 (Food) (Bossard et al., 2014), OxfordPets (Pets) (Parkhi et al., 2012), SUN397 (SUN) (Xiao et al., 2010), UCF101 (UCF) (Soomro et al., 2012), Resise45 (Resisc) (Cheng et al., 2017). Detailed task information and the batch size used for training VR are provided in Table 4.
A.2. Implement Details
All VR baseline methods are trained with consistent settings: learning rate of 40, a momentum of 0.9 (SGD optimizer (Harold et al., 1997)), and a cosine annealing scheduler (Loshchilov Hutter, 2016), over 200 epochs. Results are averaged across three random seeds. For method-specific hyper-parameters, we followed (Cai al., 2025) by using VR noise pattern with a frame size of 30 for VP (Bahng et al., 2022) and a frame size of 16 for AR (Chen et al., 2023; Tsai et al., 2020) and At (Cai et al., 2025). To ensure fairness, DVP utilized the same settings as Att.
For DVP-cls, we use the same descriptions as (Cai et al., 2025). For K-means, we use maximum iteration of 300 and the relative tolerance regarding the Frobenius norm of differences in cluster centers to be le-4. For DVP-cse, we use GPT-4o-mini (Brown et al., 2020) to generate descriptions, with the maximum token to be 50, stopped at !,, and the temperature to 0.99. We set m 20 for each cause number in DVP-cse and the attribute numbers in DVP-cls.
A.3. Generated Causes
Table 5. Top Three Causes Used by DVP-cse for Each Dataset
CAUSE I
CAUSE 2
CAUSE
AIRCRAFT CALTECH
shape and structure
size and proportions color
surface features texture
shape
CARS
front and rear design
shape and body style
wheels and rims contrast
DTD
pattern texture
coarseness/gee grarsaayssgranuulrrty spectral information shape of the petals shape and composition species and body shape
ESAT FLOWERS
spatial characteristics
flower structure texture
color
FOOD
color facial features color and lighting intensity and speed spatial characteristies
PETS
fur or coat type
SUN
spatial layout and composition
texture contextual elements texture
UCF
motion and poses spectral information
RESISC
For DVP-cse, the top three causes for each dataset are obtained by asking LLM to 'List the top three causes for classifying [Task Info.] images', with the specific results presented in Table 5. Notably, to investigate the impact of different causes on the results, we generated the top seven causes for the Aircraft, DTD, and Flowers datasets for comparison, as listed Table The results of DVP-cse in Figure and Figure 7 apply these causes. The effect of different causes will be discussed in Appendix C.6.
12

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Table Top Seven Causes Used for Aircraft, DTD and Flowers Dataset
AIRCRAFT
DTD
FLOWERS
CAUSE
shape and structure size and proportions surface features engine configuration wing configuration tail design
pattern
flower structure
CAUSE
coarseness//ranuuarity
shape of the petals color size of the Flower
CAUSE
contrast
CAUSE
directionality
CAUSE
regularity
fragrance Icaf characteristies
CAUSE
roughness
CAUSE
landing gear configuration
entropy
flowering time and growth habit
B. Appendix 2: More Theoretical Justification
B.1. Detailed Propositions and Proof
Proposition B.1. Assuming that V is variable representing the description text and Y is the variable representing the downstream label, where V E A, Y E y and A(Y S A the description subset that corresponds to class Y it can be obtained that AV & A(Y"), p(V[yT) = 0 and vV E A(Y"), = V A(Y")).
Proof. By the law of total probability (Kolmogorov, 1956), know
p(VIV") =M(iyt, V (EE((Y))) p(V E A(Y2)
+P(VIYT, V & A(Y")) p(V & A(T
(11)
For all attribute descriptions that not correspond to label y", (i.e, VV & A(y we have p(V = 0, as V is the deseription irrelevant to YT.
Besides, for attribute descriptions that correspond to y², we have
VV E A A(y))) p( A()") 1,
VV A( plV & A(YT)) 0,
(12)
By substituting the above two equation into Eq. (11), it can be obtained that AV E A(v"),
p(V[YT) = P(VIYT, V E A(YT)) x1
+ V A(Y'))) x0
=P(VIYT V E A(Y2)).
Conclusively, we get:
P(V[YT) [p(VIYT, Ve A(TT)) V A(YT) -
V A(YT)
as is stated in Proposition B.I.
Proposition B.2. Assuming (..)) is the counting function that returns the frequency of two items occurring together,
V E A, E yt are description and label variables respectively, A(YT) S A is the description subset that corresponds to class YT and a, S are values, then through MLE (Fisher, 1922), the conditional probability p(V = a|Y'T = y" v V A)) can be estimated on training set as
o --=-))
a = V. A(vT))
13

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Proof. According the definition of conditional probability, we rewrite p(V =a -y, a A(y))) as:
-V==, V" A(T"))
p(V - aaY -y". E A(y"))
p(T = -b V A(r²))
p(V - a A(y") x²)
p(V E A(yT)|TT =
Therefore, for Va A(y"):
p(V = a = = EC'EAGTT) P(V o
p(V = =y E A(T
(13)
where p(V - aaaaaa-=-- )² is value in w to be estimated. Given the downstream training set D = (xx7 (r (f") the VR method and description set A, the maximum likelihood [(w) is formulated to be
((w) II p(V - vf-),
(14)
where a; is the most similar description for sample xx! given the input DVP pattern A and description set A. For simplicity, we use Wp..- _ p(aply,) to represent p(V = aplY = yg). Thus, according to Eq. (14) any single item wp.g in w will have the maximum likelihood as
((wp.g)=
II Y' VEA
II
pVV = -Va, Y- Ng)
where (ap Mq) is the counting function that returns the frequency of matched ap and By pairs. Taking the logarithm gives:
1(wpa)=
Σ
Σ
ND((ap Mq) log app, Ng), s.t. Σ = Y'I, =VV, V44)
s.t.
YTEYT VEA
Then we introduce Lagrange multipliers Ay for constrained optimization:
l' l'(Wp.g A) ((wp.)) 21
Σ a'EAA
p(V = d'lx =yg)).
Taking partial derivatives on both sides of the equation, we get
OF
ND V4)
0p(V = ap YT = Vg)
p(V = ap VT == 319))
Therefore, p(V = Yp 34) = wDlapioo) Since EayEAP(V = apl,yT = =yg) = 1, we get Ea,ed Ea by
Nolapito
1
bg
and thus Ag -N (v²). Substituting back into Eq. (13), we get the final prediction:
Nolap, Ug)
p(V a, yy) EWEA( (Dlaaa ND(a',y,)
B.2. Relationship between Pvr and pdvp
In DVP, since A = U.= and A(yF) n Ø for # j. the total similarity between xT and A decomposes additively across partitions. Moreover, each o, optimizes only for A,, making 8, exclusive prompt responsible for aligning x. with descriptions in A.-
14

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
The integrated logits of all prompts A thus be expressed summation over all individual with respect the designated description partition Ai.
logits (xT; A (rm)))
f logita (x=;; :0 wo)))
integrated logit
(etn((fin="1)0)), (p)) 2y
ApEAA
partition-spesffic logit contribation
Given the form of Eq. (B.2), it easy to find that, by reducing A ,ō1,...,0u, 5o3 to the same 6, well as substituting w with w (i.e., agg())), DVP reverts back to standard VR.
B.3. Proof of Lemma 4.4
Proof. The standard VR method, parameterized by single prompt 6, can be viewed as a special case of the DVP model (both using a fixed wix) where all decouped prompts constrained to be identical, i.e., d1 62 ...,d.- let A tied (6.....6) 66, then the DVP logits using fix become:
8e = 6. Specifically,
[ (xT;; tied w
fix
Σ
Σ
(atip(ii(==1l),, ap) wxxx)
E,E.A
Since the partitions A; are disjoint and their union is A. this sum is equivalent to Ea,e. (Jeli(Jin ( 18), ap) wxx) This expression, when aggregated according to the rules defined in fix e.g., max(.) or avg(-) over descriptions for class yi yields the logits of the standard VR model ( A). Therefore, the set of functions representable by standard VR (parameterized by 5) is a subset of those representable by DVP (parameterized by A) when both use the same fix
Fer be the function space spanned by standard VR prompts and Favp be that for DVP prompts (both with fixed lxx,,, then Fvr S Favp. Given that 8* and A Gi represent the optimally achievable parameterization within F and Favps respectively, it is ensured that Rip (8* fix) 2 R (A fix) because minfeF. (f) S minp'eFs (f')), Fn S FA (based on Definition 4.3 and the set inclution relationship). This complete the proof.
B.4. Proof of Lemma 4.5
Proof. Assuming the hypothesis space of wprm to be @prmm and the optimized wrm e @prm to be wrmm* then for fixed A, we have:
Vi E @rrm Aevp (A, w) 2 petve (A, prrn))
(15)
We define the w in standard VR method that use max(-) in Eq. (1) to be max where max E {0, 1]14|x/> as discussed in Section 3. wpa is set to be only when Op the g-th class's attribute description with the maximum CLIP output, otherwise w 0.
Besides, when (- is used in Eq. (1), define w ways Then ways E (0, 1 1 x where way only when the p-th attribute is the description of the q-th class. As a result, fix e fa wv } for standard VR methods. For max
we have:
{
Op = arg a Ep (p = My) otherwise
mmx = P(V = a, O = =vn)
where ET(a = By) is the expectation of CLIP output à corresponding to the description a', for samples in the downstream training set whose group truth labels are Ug.
15

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
For waV, have:
P(V appY =y4)= = { P-
1 ap E A(va) otherwise
where [A(y))) is the number of descriptions for each class Vn- It can be observed that both max(-) and avg(.) special cases of PRM, which is consistent with Eq. (4) in Section 3. Therefore, m and w exist within the hypothesis space of PR.M, meaning:
w max E pprm and w a E @prm.
From Eq. (15), have:
Re (A, ) 2 r (A, wprmn)) and Rev (A, wvx) 2 Revp (A, wemm))
i (A, wixx) 1 (A, wPrm*).
B.5. Proof of Corollary 4.6
Proof. According Lemma 4.5, VA, P (A,w (xx) 2 P (A, wPmn), then we have:
R (Ai, wx) 2 AD (Ai, wPm*) R (Ag prm*),
where Aj is the optimized A for Rdvp (,, wix), while A is the jointly optimized A considering wprm* To avoid confusion, we use A and A to represent different A "s in Lemma 4.4 and Corollary 4.6.
Lemma 4.4 proves İeg (8* wix) > fdvp (A (A" wxx), i.e., Reg (0", w(xxxxx) > (Ai, Applying Lemma 4.4 in the above equation, it can be obtained that:
( R (8" wix) 2 P ( fxx 2 Pigle (D (g wrmn
= R (5* ,xxxx 2 Advp (A w prm+)
That is, Rey (8*, (8",w" (xx) 2 R (A w as A wprm+ are jointly optimzed.
C. Appendix 3: More Experimental Results
C.1. Hyper-parameters
C.I.1. CHOOSING HYPER-PAR PARAMETERS
Table Choosing Appropriate Hlper-paramette with the zero-shot accuracy
k
AIRCRAFT
CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
AvG.
27.00
93.67
66.67
55.67
51.26
84.53
86.36
91.25
66.08
70.74
61.65
68.63
27.36
93.63
66.52
55.44
52.12
84.41
86.43
91.39
66.28
70.79
61.67
68.73
27.78
93.31
66.46
54.55
51.32
83.52
86.54
91.25
65.61
70.58
61.43
68.39
27.69
93.31
66.47
54.43
50.17
82.54
86.55
91.22
65.15
70.26
61.33
68.10
For both DVP-cse and DVP-cls, the optimal value of hyper- pappm--rrrmeter is determined based the average zero-shot accuracy on the validation sets of downstream tasks under different k values in (1, 5). The results are shown in Table 7, When & = 3, the average zero-shot accuracy reached its peak. Therefore, we set & = 3 in this study.
For hyper-parameter v in DVP-cls, we selected the value from u E (1,2,3). Since the sample size and number of classes vary across datasets, we determined a -ataset-specific v using the validation set for each dataset. The detailed results are shown in Table 8.
16

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Table 8. Choosing Appropriate Hyper-parameter
AIRCRAFT
CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
38.07
87.97
56.30
87.60
96.47
89.47
81.77
88.57
66.50
77.40
85.70
41.53
89.93
59.23
93.23
96.07
91.27
83.20
89.30
69.50
82.33
88.00
3
45.27
90.07
61.17
95.20
93.77
93.00
83.17
90.40
71.03
82.30
89.03
OPTIMAL
3
3
1
2
3
3
2
3
Downstreum Task: Flowers
Downstream Task; Texture
Downstream Task: Aircraft
98.0
70.0
42.0
Accuracy (%)
Accuracy (%6
68.0
Accuracy (%)
96.0
40.0
66.0
94.0-
64.0
36.0
92.0
62.0
k
Figure 8. Impact of hyper-parameter & on DVP-cse, when & E (1,3, 5, using ViT-B ViT-B16-based CLIP as the pretrained model.
Downstream Task: Flowers
Downstreum Task: Texture
Downstreum Task: Aircraft
98.0
70.0
42.0
Accuracy (%)
Accuracy (96)
Accuracy (1)
96.0
66.0
38.0
94.0
64.0
36.0
62.0
34.0
3
9
3
9
7
9
Figure Impact hyper-parameter ls on DVP-cls, when E (1,3,5,7, 7,99, using ViT-B16-based CLIP the pretrained model.
Partition
Partition 2
Partition
Accuracy (%)
96.0
Datasets
85.0
70.0
Caltech
UCF
ESAT
Resise
94.0
82.5
Flowers
Cars
80.0
65.0
Pets
DTD
2
2
4
6
2 Hyperparameter V
Food
SUN
Hyperparameter
Hyperparameter v
Figure 10. Impact of hyper-parameter on DVP-cls, when E (1,2,3,5,7), using ViT-B16-based CL.IP as the pretrained model.
1.1.2. IMPACT OF HYPER-PARAMEN PARAMETERS
Impact of k: Figures 8 and Figure 9 illustrate the impact of different i: E (1,3,5, 7,7,9 7,9} values on DVP-cse and DVP-cls. The k setting is designed to compensate for the limited training data. For datasets with fewer classes and limited training samples, such as Texture dataset, increasing within a reasonable range (i.e., k=3, k=5) generally leads to steady improvement in accuracy. In contrast, for datasets with sufficient samples, such Flowers and Aireraft, the effect of is relatively minor.
Impact of v. Figure 10 shows the impact of different values on DVP-cls. For most datasets, accuracy increases with before stabilizing, with few exceptions. One such exception is the ESAT remote sensing dataset, which contains only 10 relatively simple classes. In this case, a larger v. implying more VPs and parameters, may lead to overfitting. Another exception the Food dataset. As mentioned in Section VR methods generally perform less effectively on this dataset, so increasing does not yield significant improvements.
17

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Weight Proportions of Different Causes after Training VR Patterss for Some Classes (Downstream Task: Texture; Method: DVP-cse)
Polka-dotted
Studded
Knitted
Fibrous
27.1%
2.1%
14.6%
20.8%
35.4%
8.3%
41.7%
60.4%
12.5%
62.5%
77.1%
pattern
coarsenessigranulangg
contrast
Figure 11. Visualization results of DVP-cse, showing weights of causes for classifying certain classes, using ViT-B16-base CLIP.
Weight Proportions of Different Causes after Training VR Patterss for Some Classes (Downstraam Task: Aircraft; Method: DVc-cse)
747-200
757-200
737-800
707-320
10.4%
10.4%
22.9%
20.8%
31.2%
25.0%
10.4%
64.6%
66.7%
60.4%
47.9%
shape and structure
size and proportions
surface features
Figure 12. Visualization results DVP-cse, showing weights of causes for classifying certain classes, using ViT-B16-based CLIP.
C.2. More Visualization Results of DVP-cse
Figure 11 shows the causes' weight distribution for specific classes in the DTD dataset, obtained from the PRM trained with DVP-cse. The results offer degree of interpretability. For instance, the 'polka-dotted' class, characterized by spotted patterns, has the highest weight for the 'pattern' cause. Similarly, 'studded', usually associated with metallic textures, shows the largest weight for "coarseness/rranuaaiii often exhibit black gaps and solid color areas from the knitting, making contrast the most important classification factor. Meanwhile, classes such 'fibrous', which have diverse characteristics, exhibit more balanced distribution of cause weights
For classes like 'knitted', which are strongly related weaving, images
Figure 12 shows the cause weight distribution for specific classes in the Aircraft dataset, obtained from the PRM trained with DVP-cse. The obtained weights also aid understanding the classification process. For instance, the 747-200, as double-deck aircraft, has a distinctive shape, making 'shape and structure' the most critical factor. The 757-200, with longer fuselage and shorter wings compared with other models, is primarily classified based on 'size and proportions' Mcanwhile, the 737-800 typically features two cabin doors and relatively large cabin windows, making 'surface features' key classification criterion.
C.3. More Visualization Results of DVP-cls
Figures 13 and Figure 14 illustrate the classes closest to the center of each description cluster in DVP-cls (i.e., sorted by the sum of each sample's distance to the cluster center). As discussed in Section 5, DVP-cls tends to group descriptions of classes with similar attributes-which might belong to the same subclass-into the same cluster.
Figure 13 presents the results on the DTD dataset, where cluster corresponds to overlayed or layered patterns, such as 'interlaced', 'frilly,, and "swirly". Cluster represents textures characterized by spots, including 'bumpy' and 'flecked'.
18

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Top Classes within Different Clusters (Downstream Task; Texture; Method: DVP-cls)
Cluster One
Cluster Two
Cluster
interlaced
7.70
bumpy 5.87
interlaced 6. 6.19
frilly
7.72
flecked 6.08
woven
6.52
swirly
7.81
blotchy
6.09
crosshatched
6.
Sorted by Distance to the Cluster Center
Sorted by Distance to the Cluster Center
Sorted by Distance to the Cluster Center
Figure 13. Visualization results of DVP-cls, showing the top 3 classes closest the description cluster centers, using VIT-B 16-based CLIP
Top Classes within Different Clusters (Downstream Task: ircraft; Method: DV DVP-cls)
Cluster One
Cluster Two
Throee
Model B200
8.08
MD-87
8.10
767-200
5.66
DHC-8-300
8.21
ERU 135
8.12
767-300
5.83
DC-6
8.28
Fokker
8.29
767-400
6.17
Sorted by Distance to Cluster Center
Sorted by Distance the Cluster Center
Sorted by Distance the Cluster Center
Figure 14, Visualization results of DVP-cls, showing the top 3 classes closest to the description cluster centers, using ViT-B B16-bd CLIP.
Cluster relates interwoven patterns, such 'interlaced', 'woven', and 'crosshatched", Figure 14 shows the results the Aircraft dataset, with consistent conclusions. For example, cluster groups all 767-x models into single cluster. This clustering effectively reflects the saoclass-based organization of different labels.
C.4. More Results Different Backbones
Table 9. Accuracy comparison of different methods trained 16-shot downstream classification tasks, using RN50-based CLIP as the pretrained model (Mean %, ours are highlighted and the highest is in bold).
METHOD
AIRCRAFT CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
AvG.
VP
16.2
80.1
44.0
43.4
59.7
$3.6
65.3
77.2
48.8
$2.0
47.7
53.5
AR
18.6
86.5
53.9
46.4
66.6
60.9
74.2
82.5
56.8
59.7
58.4
60.4
ATTRVR
20.7
89.1
53.9
54.4
72.0
74.8
75.3
88.9
59.9
63.6
58.2
64.6
DVP-cSE
23,5
89.8
55.9
57.6
74.8
88.5
66.6
DVP-CLS
22.1
89.8
54.5
80.0
75.0
88.9
61.1
Tables 1l present the detailed results of various VR methods on different datasets when CLIP adopts different image encoder architectures (RN50, RN101, ViT-B32). On average, DVP-cse outperforms the baseline method AttrV by 1.5% to 2.4%, while DVP-cls, though slightly less effective than DVP-cse, still achieves average improvement of 1 1.2% to 1.6% over Att. Notably, the performance gains from DVP are more pronounced when the pretrained image encoder uses simpler architectures, such as RN50 and RN101. The only exception among these datasets is Food. As discussed in Section 5, VR methods are less effective on this dataset, and thus DVP's decoupling-ande rewiiehhggtting framework does not improve accuracy for this task.
19

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Table 10. Accuracy comparison of different methods trained 16-shot downstream classification tasks, using RN101-based CL.IP the
pretrained model (Mean %, highlighted and the highest is in bold).
METHOD
AIRCRAFT CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
AVG.
VP
19.3
83.0
53.7
43.4
62.8
57.2
71.2
80.2
53.5
54.2
54.0
57.5
AR
19.5
89.7
62.0
46.3
70.4
60.4
78.0
84.4
58.4
60.6
60.2
62.7
ATTRVR
23.3
92.0
62.2
55.6
70.3
76.2
79.5
89.3
62.1
64.5
64.5
67.2
DVP-CSE
24.7
93.3
64.1
59.2
73.3
82.9
79.3
90.4
62.4
67.6
67.8
69.6
DVP-CLS
23.8
92.7
62.5
58.0
70.7
80.6
79.1
89.5
63.7
68.1
68.4
68.8
Table 11. Accuracy comparison of different methods trained on 16-shot downstream classification tasks, using ViT-B32-based CLIP as the pretrained model (Mean %; ours are highlighted and the highest is in bold).
METHOD
AIRCRAFT CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
AVG.
VP
24.3
92.3
58.6
54.9
85.9
71.2
75.0
86.8
61.0
67.3
73.9
68.3
AR
21.8
92.7
56.9
49.9
85.6
66.7
75.7
84.7
59,9
63.5
71.6
66.3
ATTRVR
24.5
92.0
56.6
56.8
88.6
77.8
77.2
89.8
62.8
67.9
73.9
69.8
DVP-CSE
27.3
93.4
58.1
58.8
89.1
83.1
76.8
89.8
63.5
69.0
75.4
71.3
DVP-C1S
92.9
56.5
57.2
88.5
82.5
77.0
89.2
64.2
70.5
76.0
71.0
C.5. Discussion about Training Time
Table 12, Accuracy comparison AttrVR with DesAttr and our methods with only one VP trained, using ViT-B16-based CLIP as the
pretrained model (Mean %, ours are highlighted and highest is in bold).
AIRCRAFT
CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
AvG.
ATTRVR (DESATTR)
35.9
95.6
68.2
64.4
93.8
92.4
85.7
93.0
67.7
78.6
81.8
77.9
DVP-CSE (NUM=l)
37.2
96.0
70.1
93.8
93.8
85.6
93.3
68.4
79.7
81.7
78.7
DVP-CLS (NUM=1)
36.4
95.8
69.1
94.1
93.6
85.7
70.0
78.7
Table 13. Training cost different VR methods, using the ViT-B16-based CLIP as the pretrained model and the Flowers task an example (ours highlighted ).
VP
ATTRVR
DVP-CSE (NUM=1)
DVP-CLS (NUM=1)
PARAMETER NUMBER TRAINING TIME FOR EACH EPOCH (S)
69.8K
39.9K
39.9K
39.9K
39.9k
2.97=0.02
2.85+0.03
2.83+0.00
2.87=0.02
2.88+0.04
TRAINING TIME IN TOTAL (MIN)
9.78+0.07
9.44+0.05
9.54+0.04
9.70=0.03
9.66 # 0.07
A potential limitation of DVP is the increased training time required due to the larger number of VPs that need to be optimized. However, since each VP is trained independently, the process can be parallelized, minimizing any significant increase in overall training time. Additionally, the time cost of computing the PRM primarily involves matrix operations, which do not require gradient calculations. Compared with the time-consuming forward pass of CLIP, this overhead negligible.
Even under the constraint that DVP must use only VP as the baseline method for a fair comparison, it can still achieve improvements over the baseline with almost no additional time overhead. Table 12 compares AurVR and DVP when only single VP is used, while Table 13 shows the runtime of various VR methods. For fairness, the number of descriptions is kept consistent, and only one VP is used for each method. The results demonstrate that both DVP-cse and DVP-cls can deliver an average accuracy improvement with negligible extra computational cost.
C.6. Exploring Impact of Different Causes
Figure 15 demonstrates the impact of using a single cause DVP-cse on the classification results for Flowers. Some causes, such as flower structure, size, and leaf characteristics, significantly contribute to the success of reprogramming. These causes are well-suited for reprogramming because they are visually distinctive, making them easier to optimize through
20

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Results of the Single Decoupled VR Pattern trained with a Single Cause (Downstream Task: Flowers)
Samples
95
Different Causes
flower structure
94
Accuracy (%)
shape of petals
93
size of the flower
fragrance
92
leaf characteristics
flowering time and growth habit
91
Figure 15. Comparison results of different for classifying Flowers applying DVP-ese with single cause, using ViT-B16-hased CLIP.
Results of the Single Decoupled VR Pattern trained with Single Cause (Downstream Task: Texture)
Samples
68
Different Causes
pattern
66
coarseness/gramulariyy
Accuracy
contrast
64
directionality
regularity
62
roughness
entropy
60
Figure 16. Comparison results of different causes for classifying DTD applying DVP-ese with single cause, using ViT-B16-bascr CLIP.
Results of the Single Decoupled VR Pattern trained with Single Cause (Downstream Task: Aircraft)
Samples
38
Different Causes
shape and structure
37
Accuracy (%)
size and proportions
surface fcatures
36
engine configuration
wing configuration
35
tail design
landing gear configuration
34
Figure 17. Comparison results of different causes for classifying Aircraft applying DVP-cse with single cause, using ViT-B16 -----ed CLIP
VP patterns, and they are causally linked to correct classification. Conversely, causes like color, fragrance, and flowering hine/growth habit are less effective for reprogramming. Color is less suitable because individual flowers within the same class may exhibit color variations, reducing its contribution accurate classification. Meanwhile, fragrance and flowering
21

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
DVP-cse
DVPlite
VP and Corresponding Causes
Input Image
Model Reprogramming
Input Image
VP
Flower Structure
Reprogramming
Model
VP
Shape the Petals
VP
Color
VP
Size of the Floowr
Figure Pipeline original DVP-cse and DVPlite under setting of four decoupled causes, using Flowers dataset an example. For different causes, DVP-cse requires training multiple separate VPs, multiplying the parameters, In contrast, DVPlite decouples the VPs into up, down, left, and right directions, assigning them to different causes, which maintains parameter count comparable to other VR methods.
time/growth habit are not visual features, making them inherently unsuitable for VR methods.
Figure 16 and Figure 17 respectively illustrate the influence of different causes on classification results for the DTD and Aircraft datasets. In Figure 16, it is evident that most causes, except for entropy, yield similar results, while entropy leads to noticeably poorer VR performance. This is because entropy, as cause, lacks direct connection visual information, making it difficult optimize classification through VR. In contrast, causes such as pattern, contrast, and regularity directly tied to visual features, facilitating effective reprogramming. Figure 17 highlights that the most suitable causes for VR in the Aircraft dataset are shape and structure, engine configuration, and wing configuration.
In summary, the most effective attribute descriptions for guiding VR meet two key criteria: (1) they are related to visual information, and (2) they have a causal relationship with accurate classification.
C.7. DVPlite: A Potential Method for Parameter Reduction
When aiming to enhance the accuracy as DVP does without increasing the parameter overhead, we propose DVPlite as a potential alternative. enables decoupling-and rndweeeghingrg while keeping the parameter number consistent with that of other VR methods.
VR pattern consists of trainable parameters arranged alongside the four edges of the image (i.e., padding). In DVP, each VR pattern is trained independently, resulting in substantial increase in the total number of parameters. The proposed DVPlite method in this section divides the VR pattern into four independent parts, with a total number of trainable parameters equal to that of ARt.VR. Each part is positioned at the top, bottom, left, and right peripheries, as is shown in Figure 18.
Table 14, Accuracy comparison of our DVPlite and the best best-performing baseline method Att trained 16-shot downstream classification task, using ViT-B ba-ed CLIP as the pretrained model (Mean %, ours highlighted and the highest is in bold).
AIRCRAFT
CALTECH
CARS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
AvG.
ATTRVR (BASELINE)
36.6
95.7
68.3
65.6
93.8
92.9
85.9
93.3
69.6
79
82.6
78.5
DVPLITE
39,3
95.9
71.4
66.5
93.8
95.2
85.8
93.4
71.6
81
83.6
79.8
Table 15. A comparison of parameter numbers for different methods and their average accuracy across 11 datasets, using ViT-B16-basee CLIP as the pretrained model (Mean % ours are highlighted.
VP
AR
AFTRVR
DVP-CSE
DVP-CLS
DVPLITE
PARAMETER NUMBERS AVERAGE ACCURACY OVER I1 TASKS
0.07M
0.04M
0.04M
0.12M 80.
0.04-0.12M 79.7
0.04M 79.8
74.4
76.5
78.5
Accuracy comparisons of our DVPlite and the best- pest-rerfonnng baseline method Att are shown Table 14, and the parameter numbers are shown Table 15. In the DVPlite experiments presented here, the assignment of which specific VP was applied which spatial location was determined randomly (and kept fixed for that run). DVPlite then implemented
22

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
under our proposed two-stage DVP-PRM framework, with the four parts trained independently.
Therefore, the core DVP-PRM principles yield benefits without increasing the parameter count. DVPlite outperformed the Att baseline with an identical parameter budget, demonstrating the effectiveness of the decoupling-and -nd---eeigghiiingg strategy itself. However, investigating structured or learnable assignments of VPs to spatial locations instead of random assignments might be an interesting open question that deserves formal and systematic exploration future studies.
C.8. Applying DVP-PRM to Different VR methods
Table 16. Results of applying our DVP-PRM framework to various VR methods, including VP and AR (for AR, results equal that of DVP-cse and DVP-cls), trained on 16-shot downstream classification task, using WTT-B16-basee CLIP as the pretrained model,
AIrcRAFT
CALTECH
CArS
DTD
ESAT
FLOWERS
FOOD
PETS
SUN
UCF
RESISC
AvG.
VP+ + OURS-CSE (%)
6.2
+ 1.2
+.3
+.0
+.66
-922
0.2
1.1
2.4
2.6
2.4
3.1
VP + OURS-CLS (%)
6.3
+ 0.8
+ 1.2
+ 2.0
+.00
+ +8.4
+ 0.2
1.1
2.5
+35
+ 2.7
+ 2.6
AR + OURS-CSE (%)
8.6
+ 0.7
+.55
+.77
+.55
+9.5
+ 0.4
+.44
+322
+366
+ 3.0
+.6
AR + OURS-CLS (%)
7.0
+ 0.5
+.88
+ +3.5
0.7
+9.1
+ 0.5
0.6
+.2
+ 3.9
+ +
+31.1
Given the existence of various VR methods, including watermarking-baan based (i.e. VP(Bahng et al., 2022)) and padding-based (i.e. AR (Tsai et al., 2020)), we apply the proposed DVP-PRM framework to both types to evaluate its performance. Notably, applying the framework to the padding-based VR setting corresponds to the DVP-cse and DVP-clss results presented in this paper. Performance improvements achieved by incorporating our module into different VR methods are summarized in Table 16. These results suggest that our method is effective for both watermarking based and aadding-based VR, consistently providing performance gains across these different approaches.
C.9. Training Results Under Sparse Data Conditions
Table 17. Accuracy comparison of different VR methods with fewer or more training samples (i.c., 1-, 4-, 8-, and 32-shot tasks), using 11--B16-asedd CLIP as the pretrained model and the Aircraft task as an example (Mean %tSSt %, ours are highlighted and the highest is in bold).
VP
AR
AUTRVR
DVP-CSE
DVP-CLS
1-SHOT
24.1
15.1
31,5+0.2
28.1
29.4e0.5
4-SHOT
27.520.2
27.720.4
33,410.6
32.310.1
32.5
8-5HOT
29.5+0.3
10..
35. In0.3
36.4+0.1
35.9-0.5
16-SHOT
32.1
31.7=0.3
36.620.3
40.3:0.2
38. 28.720.4
32-SHOT
34.5#0.4
34.2+0.7
38.4+0.3
43.1
41.2+0.5
To evaluate the performance of DVP under varying amounts of training data, we conducted experiments on the Aircraft dataset with 4- 8-, 16-, and 32-shot settings. The results are presented in Table 172.
can be observed that under extremely limited training conditions, such as the 1-shot and 4-shot scenarios, DVP performs worse than the baseline methods. This is attributed the fact that the MLE in the PRM module becomes more reliable with larger amounts of data. The suboptimal performance under very low-data conditions highlights one of the limitations of DVP.
However, the performance improvement of DVP becomes increasingly evident when the number of training shots exceeds 8 for the Aircraft dataset, and is particularly pronounced under the 32-shot setting. This suggests that as the amount of training data increases, DVP can better leverage its modeling capacity, resulting in more substantial gains.
D. Appendix 4: Problem Setup & Notation
D.I. Problem Setup Comparison
We clarify the differences between the problem this work focuses and another active research theme to avoid confusion.
Note that conducted additional experiments and correct the 8-shot and 32-shot results reported in the rebuttal, but these corrections do not affect conclusions presented during rebuttal.
23

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
Visual Reprogramming (This Work). The focus of this paper address the challenge of adapting CLIP, pretrained aligned image-text pairs, downstream classification tasks where input dimensions mismatch the original model (dr # dg). Following established visual reprogramming protocols, we learn set of visual prompts (VPs) A ói,.... 8, that perturb input images to align them with CLIP's pretrained feature space. Formally, given a downstream dataset D = {(=] v ) with images off XT and labels of eT,, optimize A to minimize the cross-entropy loss:
A arg min
p(y fulafia))
where [in ; A) pad(x*) + A pads to match CLIP's expected dimensions and applies trainable visual prompts A.
Text Prompt Tuning. In contrast, text prompt tuning methods like CoOp (Zhou et al., 2022a) adapt pretrained CLIP by modifying its text imput space. Instead of perturbing images, these studies aim to optimize set of continuous context vectors C ; prepended to class names, forming prompts such ""c1 C. (Class Info]." For class y, its embedding becomes
Sixt y) - Stxt (.i. Stnkenize (y))).
where frokenize (v) tokenizes the class name y, The learning objective maximizes the alignment between image embeddings fing (x=") and adapted text embeddings (C,
N
C*
arg min C
Σ
log plg
N
1=1
D.2. Notations
We then provide a list of key mathematical notations used throughout the paper to ensure clarity and consistency.
D.2.1. GENERAL CONCEPTS REGARDING CLIP MODEL.
Table 18. General Concepts and Notation
Symbol
Description
CLIP
Contrastive Language- Image Pre- Pre-training model (Radford et al., 2021).
fing
CL.IP's image encoder.
fest
CLIPP's text encoder.
x
Input image space for the pre-trained CLIP model (i.c., source domain images). X C Rds. Text space containing all textual descriptions.
V
Z
Shared embedding space for images and text.
v
textual phrase or description from
Input image space for downstream task (target domain images). XT C Rd
yT
I Label space the downstream task.
T
An image from the target domain T A label from the target label space y/T Dimensionality of source domain images.
TT
ds
dr
Dimensionality of target domain images.
(a V)
CLIP similarity score between image n and text V. Defined in Eq. (1). Temperature parameter in CLIP's similarity computation.
A
The complete set of textual descriptions used for all classes in downstream task. Suhset of A containing m descriptions specifically for class y²
A(v²))
m
Number of textual descriptions per class.
[hogiza ( A))rr
Logit for class y given image z". computed using standard CLIP when dr ds. Generic aggregation function (e.g., max(-), avg(-) over description similarities. The cardinality of a set, ie.., the number of elements in a set.
agg(-) 1.1
24

---

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
D.2.2. STANDARD VISUAL REPROGRAMMING (VR)
Table 19. Standard Visual Reprogramming Notation
Symbol
Description
fin (1)))
Input transformation function in VR, fin fin pad(x") 6. Zero-padding function to make x² dimensionally compatible with fing.
p-d(.)
The single, trainable visual prompt vector tensor in Rds) in standard VR.
[Faga (x:: 6, A)
Logit for class y given image x" and prompt in standard VR. Defined in Eq. (1).
Py 61,8,
Normalized probability for class y" in standard VR, obtained via softmax over flggts- Row vector (1 [A)) of CLIP similarities between fin (x." (8) and all descriptions in A. Row vector D () of class logits.
Ma
My
General reweighting matrix (I.A) x (>T)) relating a to y via a., v2,6w-
wapa fix
Element of w, representing the contribution of the p-th description to the q-th class logit.
Fixed reweighting matrix used in standard VR, determined by the choice of agg(-) (egg. max(.) or avg(.)).
D
Downstream training set, consisting of N image-labee label pairs: {xx] u3 )
N
Total number of samples in the training set D.
D.2.3. DECOUPLED VISUAL PROMPTS (DVP) FRAMEWORK
Table 20. Decoupled Visual Prompts (DVP) Framework Notation
Symbol
Description
A
Set of e visual prompts, i.c., A {61, . 6.).
8
The i-th decoupled visual prompt, specialized for description partition A,,
A
The full description set.
E
The full set of text embeddings for descriptions in A.
A
The i-th partition of the full description set A.
&
The of text embeddings corresponding descriptions in partition A..
Number of decoupled visual prompts and description partitions.
DVP-ese
DVP variant where partitions A, are formed based on explicit semantic causes from an LLM. DVP variant where partitions As are formed by unsupervised clustering of description embeddings. Probabilistic Reweighting Matrix: The learned (I-AI [D" matrix the DVP framework. Entry of PRM, estimated as P(V a, Vo V E A((y))).
DVP-cls
PRM (WPm
ND(a, v)
Co-occurrence count of description and class y training set D. Used for PRM estimation. Set of top-k descriptions in A most similar to [x,] [A) for image x²..
X(xx,, k))
Number of top similar descriptions considered for smoothed PRM counting.
Po T 66, pm As)
Probability derived from ( (x"))
Ulogita(e (a"", T;;;; whm))) 2
Integrated logit for class vi DVP, combining all prompts and PRM wPrm.
(y 1z A, Prm AA
Final probability for class y² DVP, from integrated logits.
25


Reviews:
[{"review_id": "qaSSRIU7dq", "reviewer": "Reviewer_MNe5", "summary": "This paper proposes a visual reprogramming method for CLIP. The existing methods utilize a single visual prompt for all descriptions, but a single prompt may not enough to capture diverse aspects of class descriptions. The authors propose to have multiple visual prompt for each description (decouple), and integrate the outputs from each visual prompt with a probabilistic reweighting matrix. The proposed method was evaluated on various downstream classification tasks and obtain the improved results compared to other visual reprogramming methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "F (Feedback); R (Response)\n\n**F1.** Pros and cons of visual reprogramming (VR)\n\n**R1:** Key difference: VR applies transformations only to input and/or output, while other PEFTs adapt within internal architectures, leading to 3 pros of VR:\n- VR untouches any parameters and architecture, is applicable to *any* models, crucial for cases requiring entire model preservation to avoid catastrophic forgetting, copyright issues, or working with black-box APIs; in contrast, other PEFTs are architecture-dependent, e.g. VPT was sepcifically developed for ViT.\n- In VR, #trainable params depends only on *input data dims*, leading to lower training overhead compared with other PEFTs (e.g. LoRA, VPT) whose parameters scale with the pretrained model.\n- VR is orthogonal to other PEFTs and can be combined with them.\n\nSo, VR has a unique position within PEFT the spectrum, offering complementary values to other methods.\n\n**F2.** Motivation experiment regarding the 'height' of flowers\n\n**R2:** We replaced the 'height' with 'color' descriptions (i.e., 'Artichoke petals exhibit a mix of soft purple, lavender, and green tones') and still observed similar results, confirming the reliability of our motivation. In the revision, we will refine results in Fig. 2 by replacing 'height's with 'color's.\n\n**F3.** Proof of Lemma 4.2\n\n**R3:** Thanks for pointing it out. We can analyze the hypothesis spaces induced by VR and DVP methods, which is the standard way to compare the potential performance of models with different capacities. Due to the character limit, we provide a proof sketch and will present the completed proof in the next version.\n1. Define the hypothesis spaces representable by VR $\\mathcal{H}\\_{\\rm VR}$ parameterized by $\\delta$ and DVP $\\mathcal{H}\\_{\\rm DVP}$ parameterized by $\\Theta$, mapping inputs to outputs under fixed $\\omega\\_{\\rm fix}$.\n2. $\\mathcal{H}\\_{\\rm VR} \\subseteq \\mathcal{H}\\_{\\rm DVP}$ as DVP has a larger parameter space and it specializes $\\delta\\_i$ for description partitions $\\mathcal{A}\\_i$, offering greater flexibility.\n3. We conclude that optimization over the richer space allows DVP to find a function $f$ within $\\mathcal{H}\\_{\\rm DVP}$ that fits training data $\\mathcal{D}$ at least as well as the best function found within $\\mathcal{H}\\_{\\rm VR}$. \n\nThus, the minimum empirical risk $\\hat{R}^{\\rm dvp}\\_{\\mathcal{D}}$ searched over $\\mathcal{H}\\_{\\rm DVP}$ must be less than or equal $\\hat{R}^{\\rm vr}\\_{\\mathcal{D}}$.\n\n**F4.** More experiments\n\n**R4:** We include more PEFTs (VPT, Adaptor) and mark the number of trainable parameters as reqeusted. Due to space limitations, we report the average accuracy across all 11 datasets. `DVPlite` is a new DVP variant with reduced parameters by decoupling the padding pattern of visual reprogramming (top, bottom, left, right) while maintaining the same number of parameters as the baseline. \n|      |   PEFT-VPT   | PEFT-Adaptor | VR-AttrVR | VR-DVP (ours) | VR-DVPlite (ours) |\n|-|-|-|-|-|-|\n|           params         |  0.04M  |  <0.40M  |  0.04M |    0.12M   |      0.04M      |\n|   RN50-CLIP (11 tasks)  | Invalid |   55.9  |  64.6  |    66.6    |        66.1       |\n| ViT16-CLIP (11   tasks) |   78.8  |   76.4  |  78.5  |    80.1    |       79.8      |\n\nRegarding the reweighting matrix (PRM): we did not apply it to baselines, as PRM is one of the contributions of *this study*. In the ablation study, we have evaluated the standalone effect of PRM to isolate its utility.\n\nWe thus observe two key merits of our method:\n- Works with all architectures (RN, ViT-based), but VPT is incompatible with RN-based models.\n- Improve accuracy without necessarily increasing parameters.\n\n**F5.** Application in other tasks/multi-modal LLMs\n\n**R5:** While we follow previous VR studies on classification, the idea of DVP-PRM can extend to other tasks because CLIP learns general-purpose visual and text representations, suitable for tasks like segmentation and detection, and DVP *does not* fundamentally change the CLIP prediction pipeline.\n- In segmentation, DVP can motivate methods operating at the pixel/patch level with decoupled prompts specific to different semantic elements (boundaries, textures), while using PRM to generate dense prediction maps.\n- In detection, DVP can integrate into a two-stage detector, where decoupled prompts first generate region proposals focusing on objectness and scale variations, then region-specific PRM refines classification and localization.\n\nBesides, the workflow of DVP is similar on MLLM and CLIP, with the only difference being the use of instructions for downstream tasks.\n\nWe appreciate these extensions are promising directions *beyond the scope* of this paper. In this paper, we focuse on extending single visual prompt capabilities and provide interpretability for reprogramming. Experiments on these applications can be explored in future work.\n\n---\nThank you for your time! If our response has resolved your concerns, we would appreciate a higher score.", "claims_and_evidence": "Motivation: I agree that a single visual prompt is not appropriate to model all the descriptions for classification. However, Figure 2 may not appropriate to fully show the necessity of multi visual prompts because \"Height\" description score might be decreased due to the absence of visual cue for height in the example images. Even as human, we cannot say that the flower in the image are growing up to 4 or 5 feet tall. I recommend the authors to include the clearer sample that the description, which is clearly shown in the image, has decreased score after reprogramming.\n\nNecessity of visual reprogramming: Visual programming is an interesting research topic, but I want to see the clear pros and cons compared to other PEFT methods (e.g., LoRA, visual prompt tuning, adapter, ...). I can easily come up with some cons of the visual reprogramming (e.g., fixed input resolution, requiring LLM heuristic description, ...) but not for pros of visual reprogramming. The clear comparison with other PEFT methods will be appreciated.", "methods_and_evaluation_criteria": "Yes, the proposed methods sound reasonable and the evaluation criteria is acceptable.", "theoretical_claims": "I checked the proofs for theoretical claims. I think the proof for Lemma 4.2 has some flaws. The main conclusion of this proof should show that \"optimizing flexible models will result in lower empirical risks than identical models\", but the authors use them as assumption (Lines 736-737). For more complete proofs, the authors should analytically proof the assumption as well.", "experimental_designs_or_analyses": "Baselines: I recommend the authors to include baselines from other family of PEFT methods, like visual prompt tuning, LoRA, adapters, etc, with presenting the number of learnable paramaters. \n\nFair comparison with baselines:\n- To show the effectiveness of decoupling prompts, the reweighting technique should be applied on baselines in Table 1 (i.e., not just max/avg across all the descriptions, but learning the weight for each description).\n\n- The authors should present the comparison of learnable params in Table 1. I guess the proposed method has more number of learnable parameters than other reprogramming methods because the proposed method has multiple visual prompts for each description.", "relation_to_broader_scientific_literature": "The motivation is interesting, but as noted above, I don't see the advantages of visual reprogramming compared to other family of PEFT methods.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Please see above", "other_comments_or_suggestions": "As done in visual prompt tuning paper [ref1], it would be great if the proposed method can also be applied on other visual tasks like segmentation, detection, etc.\n\n[ref1] Visual Prompt Tuning", "questions_for_authors": "It can be out of scope, but can this method be applied to multi-modal LLM?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kkCYhO6O2r", "reviewer": "Reviewer_Gm1j", "summary": "This paper proposed the Decoupled Visual Prompts (DVP) to improve the limited learning capacity of the single VP. Explicit causes and unsupervised clusters are grouped with descriptions and decoupled-and-reweighted VPs are trained to improve the final performance. Experiments show better results, compared with the single VP.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "M (Methods And Evaluation Criteria); E (Experimental Designs Or Analyses); R (Response)\n\n**M1/E1.** Increased computation resource caused by DVP\n\n**R-M1/E1:** Thank you for your suggestion! We have included the parameter numbers for various methods in the following table and will incorporate it into the main text in the next version.\n\n|                                |   VP  |   AR  | AttrVR | DVP-cse (ours) |   DVP-cls (ours)  | DVPlite (ours) |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|             params             | 0.07M | 0.04M |  0.04M |  0.12M  | 0.04-0.12M |  0.04M  |\n| Average accuracy over 11 tasks |  74.4 |  76.5 |  78.5  |   80.1  |    79.7    |   79.8  |\n\nIndeed, using DVP-cse or DVP-cls does increase the number of visual prompts, which will multiply the trainable parameter numbers. \n\nHowever, in response to your feedback, we proposed DVPlite--a decoupling method that keeps the parameter numbers the same as a single visual prompt--to test the effectiveness of our DVP-PRM framework. DVPlite decomposes the original padding-based visual reprogramming pattern into four separate components (top, bottom, left, and right), training them as independent visual prompts.\n\n|              | params   | Aircraft | Caltech | Cars | DTD  | ESAT | Flowers | Food | Pets | SUN  | UCF | Resisc | average |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| AttrVR-baseline |0.04M| 36.6     | 95.7    | 68.3 | 65.6 | 93.8 | 92.9    | 85.9 | 93.3 | 69.6 | 79  | 82.6   | 78.5    |\n| DVPlite  |0.04M       | 39.3     | 95.9    | 71.4 | 66.5 | 93.8 | 95.2    | 85.8 | 93.4 | 71.6 | 81  | 83.6   | 79.8    |\n\nAs shown in the table above, DVPlite achieves performance comparable to DVP-cse or DVP-cls *without increasing the parameter numbers*, demonstrating its efficiency compared to the baseline method AttrVR.\n\n**M2.** The feasibility of using a fully connected layer\n\n**R-M2:** Thank you for your question! Replacing the Bayes rule computation of probabilistic reweighting matrix (PRM) with a single-layer neural network is theoretically feasible but presents two major issues:\n\n- *Significant increase in parameter count* – Assuming each class has $m$ descriptions and the total number of classes is $c$, the additional trainable parameters would be $mc^2$. For large-category tasks like SUN397, this results in two orders of magnitude more parameters than a visual prompt.\n\n- *Inability to Preserve PRM’s Sparsity* – As shown in Equation 5, PRM is a sparse matrix, meaning each class is only associated with its own descriptions, with zero weights for others. Using a fully connected layer to approximate this sparse matrix would likely yield suboptimal results.\n\nFor these reasons, we compute PRM using the Bayes rule instead of a trainable linear layer.\n\n**E2.** Different results of the baseline method\n\n**R-E2:** Thank you for your question. For the baseline results, we directly used the results provided in the paper of AttrVR. The difference in the average value is because AttrVR uses 12 datasets, while our study focuses on 11 finer-grained downstream classification tasks to better figure out the functionality of each visual prompt, leading to a different average accuracy.\n\n---\nThank you again for your time!", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. The proposed methods are reasonable. Using the decoupled VP set should be better than using the single VP.\n\nWeaknesses or questions:\n1. Using the decoupled VP set would increase the runtime and computation resource. This should be analyzed through theory and experiments.\n2. The probability reweighting matrix is computed by the bayes rule on the training set. But is it possible to train an MLP to replace the matrix?", "theoretical_claims": "No Theoretical Claims.", "experimental_designs_or_analyses": "Yes.\n\nWeaknesses or questions:\n1. Please show the FPS and Number of Parameters of each method in Table 1.\n2. Why the results of ATTRVR in this paper is different with that in the ATTRVR paper?", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XQu6M6TtLn", "reviewer": "Reviewer_naZq", "summary": "The authors studied the visual prompting problem of CLIP and proposed a new method DVP.\nDVP focuses on introducing different causes to generate attributes on the text side and assigning an optimizable VP to each cause.\nIn addition, the authors also proposed a reweighting matrix strategy to integrate all causes to generate the final prediction.\nThe experimental results prove the effectiveness of DVP.\n\n## update after rebuttal\nThe authors' reply addressed my concerns. I hope the authors can improve some of the descriptions to alleviate readers' confusion. I decided to increase my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "T (Theoretical Claims); E (Experimental Designs Or Analyses); W (Weaknesses); R (Response)\n\n**T/W1:** Issues of mathematical symbols\n\n**R-T/W1:** Thanks for the feedback on clarity and notation! We acknowledge these concerns, will supplement a notation table and smooth out the presentation in the next version.\n\nTranspose in Equation 2: You are correct. $M_y$ is defined as a column vector, and the RHS in Equation 2 should have required a transpose to ensure dimensional consistency. We will add the transpose symbol to correct this.\n\nFor the relationship of $c,m$ and $v$. \nWe apologize for the confusion. $c$ indeed represents the number of classes.\nAs defined in the text, $m$ is the number of descriptions per class (L152, first column), and $v$ is the total number of visual prompts (L160, second column).\nImportantly, $v$ is a hyperparameter chosen independently of $c$ and $m$.\nWe will clarify this in the next version.\n\n**E:** Not providing the source code\n\n**R-E:** We kindly note that we have already included **an anonymous code link in the abstract section** of our submission. Please feel free to check it out.\n\n\n**W2:** More results of baseline methods with our module\n\n**R-W2:** We will first explain the difference between the method 'w/o CSE' and the baseline method AttrVR in Table 1. AttrVR follows the pipeline in the original paper, using a 'knn module' proposed by the authors and two types of attribute descriptions. However, the 'w/o CSE' in the ablation study is equivalent to the scenario where DVP-cls contains only one visual prompt, meaning it only utilizes the probabilistic reweighting matrix (PRM) module proposed in this paper and relies solely on the reweighting the descriptive attributes from AttrVR. In summary, \"w/o CSE\" is primarily used to explore the results of reweighting descriptions without decoupling.\n\nNext, following your suggestion, we present the performance gains (+%) achieved by integrating our module with other baseline methods. As shown in the table below:\n\n|                | Aircraft | Caltech | Cars  | DTD   | ESAT  | Flowers | Food  | Pets  | SUN   | UCF   | Resisc | average |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| VP + ours-cse  | + 6.2    | + 1.2   | + 4.3 | + 4.0 | + 0.6 | + 9.2   | + 0.2 | + 1.1 | + 2.4 | + 2.6 | + 2.4  | + 3.1   |\n| VP + ours-cls  | + 6.3    | + 0.8   | + 1.2 | + 2.0 | + 0.0 | + 8.4   | + 0.2 | + 1.1 | + 2.5 | + 3.5 | + 2.7  | + 2.6   |\n| AR + ours-cse  | + 8.6    | + 0.7   | + 4.5 | + 4.7 | + 0.5 | + 9.5   | + 0.4 | + 0.4 | + 3.2 | + 3.6 | + 3.0  | + 3.6   |\n| AR + ours-cls  | + 7.0    | + 0.5   | + 2.8 | + 3.5 | + 0.7 | + 9.1   | + 0.5 | + 0.6 | + 3.2 | + 3.9 | + 2.8  | + 3.1   |\n\nIt can be observed that our DVP-PRM module consistently achieves significant improvements across all baseline methods.\n\n\n**W3:** Why does each class use all descriptions instead of only the descriptions generated by themselves?\n\n**R-W3:**  Thank you for raising your concern.\nYour understanding is correct, the text descriptions are initially generated for specific classes based on causes.\nCrucially, during the integration step detailed in Equation 5, each class $y_q$ only considers the description generated specifically for it. This is **ensured by the summation condition** $a' \\in \\mathcal A (y_q)$ in Equation 5, which restricts the aggregation to the set of descriptions $\\mathcal{A}(y_q)$ associated solely with class $y_q$.\nThe computation thus aligns precisely with the intuition that class-specific descriptions should be used for each class.\n\nWhile Fig.3 visualizes the PRM matrix (which is sparse in fact, with most values being zero), the core class-specific computation is defined by Equation 5. A detailed explanation of this computation can be found in Appendix B.2. We hope this clarification resolves your concern.\n\n---\n\nThank you for your time! If you have any further questions, please feel free to ask—we’d be happy to provide more clarification. If our response has resolved your concerns, we hope you might reconsider your rating.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, I think the proposed method makes sense for the visual reprogramming problem.", "theoretical_claims": "I checked the equations in Sec. 3.\n\nI think the transpose of the matrix $M_y$ is wrongly labeled. (in Eq. 2)\n\nI think $c$ is the class number, and what is the relationship between $m$ and $v$? This confuses me.", "experimental_designs_or_analyses": "Yes, I checked the results and settings, but the authors did not provide the source code.", "relation_to_broader_scientific_literature": "The authors studied the visual prompting problem of CLIP and proposed a new method DVP.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The experimental results are very rich, making the conclusions very convincing.\n2. The structure of the submission is well-organized.\n\nWeaknesses:\n\n1. Too many mathematical symbols in Sec. 3 lead to poor readability, making it difficult for readers to directly understand the proposed method. And some symbols errors in Theoretical Claims. \n2. In the ablation study, what is the relationship between 'w/o CSE' and baseline methods in Table 1? The authors should provide the results of some baseline methods with the proposed module to prove the effectiveness of the method.\n3. I think the writing of Sec. 4.1 is not clear. In my understanding, there are some causes at the beginning, and then the authors generate corresponding text descriptions for each class based on the LLM and each cause. So in the integration, why does each class use all descriptions from all classes instead of only the descriptions generated by themselves?", "other_comments_or_suggestions": "please refer to Other Strengths And Weaknesses.", "questions_for_authors": "please refer to Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KUuJ9l6YTw", "reviewer": "Reviewer_3hP9", "summary": "This paper introduces a new method for model reprogramming on CLIP by decoupling visual prompts (DVP) to improve performance in downstream classification tasks. Instead of training a single visual prompt for all class descriptions (which may lead to biases and suboptimal learning), the paper proposes: 1) Decoupling: Different visual prompts are trained separately based on specific explicit causes or unsupervised clusters. 2). Reweighting: A probabilistic reweighting matrix (PRM) is used to measure and adjust the contribution of each visual prompt. The approach reduces empirical risk, improves interpretability, and outperforms existing methods on 11 downstream datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W (Weakness); R (Response)\n\n**W1:** Applicability to other VLMs and other tasks\n\n**R1:** Thank you for your advice! While we follow previous VR studies on classification, the idea of DVP-PRM can extend to other tasks, because CLIP learns general-purpose visual and text representations, suitable for tasks like segmentation and detection, and DVP *does not* fundamentally change the CLIP-based prediction pipeline.\n\n- In segmentation, DVP can motivate methods operating at the pixel/patch level with decoupled prompts specific to different semantic elements (boundaries, textures), while using PRM to generate dense prediction maps.\n- In detection, DVP can integrate into a two-stage detector, where decoupled prompts first generate region proposals focusing on objectness and scale variations, then region-specific PRM refines classification and localization.\n\nWe appreciate that these extensions are promising directions *beyond the scope* of this paper. In this paper we focus on extending single visual prompt capabilities and provide interpretability for reprogramming. Experiments on these applications can be explored in future work.\n\n**W2:** Training complexity and low-data regimes\n\n**R2:** Thank you for the feedback.\n\nRegarding training complexity, we acknowledge that DVP-cse and DVP-cls could increase trainable parameters compared to a single visual prompt.\n\nWhile we argue that the increased complexity is not that significant, we further introduce DVPlite, a variant specifically designed to address this potential efficiency concern.\nDVPlite decouples the padding-based visual prompt into four separate components (top, bottom, left, and right) *trained independently*, while keeping the total number of *trainable parameters equivalent* to the baseline (AttrVR).\n|              | params   | Aircraft | Caltech | Cars | DTD  | ESAT | Flowers | Food | Pets | SUN  | UCF | Resisc | average |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| AttrVR-baseline |0.04M| 36.6     | 95.7    | 68.3 | 65.6 | 93.8 | 92.9    | 85.9 | 93.3 | 69.6 | 79  | 82.6   | 78.5    |\n| DVPlite  |0.04M       | 39.3     | 95.9    | 71.4 | 66.5 | 93.8 | 95.2    | 85.8 | 93.4 | 71.6 | 81  | 83.6   | 79.8    |\n\nAs shown in the table below (will also be included in our revision), DVPlite maintains the same parameter count (0.04M) as AttrVR while achieving improved average performance (79.8% vs. 78.5%), demonstrating the benefits of our decoupled framework can be achieved *without necessarily increasing parameter complexity*.\n\nRegarding how probabilistic reweighting matrix (PRM) performs in low-data regimes, we clarify that **all experiments presented in this paper were conducted in a challenging 16-shot setting**. The strong performances across downstream datasets under this condition show PRM's effectiveness even when training data availability is already limited.\nMoreover, as detailed in the paper (L232), PRM incorporates a top-$k$ mechanism that leverages information aggregated from the $k$ most relevant descriptions when calculating the reweighting factors, mitigating potential data scarcity issues and ensuring the stability of probability estimations.\n\n**W3:** Not comparing with prompt tuning methods\n\n**R3:** Thank you for raising this point. We acknowledged CoOp/CoCoOp are important methods in prompt tuning for VLMs but did not compare with text prompt tuning methods (e.g., CoCoOp) for the following reasons:\n\n- Our work focuses on **visual reprogramming** (VR), where modifications are made directly to the input image space, without altering model parameters or internal text representations. In contrast, CoOp/CoCoOp perform **text prompt tuning**, learning continuous *embeddings* that interact with CLIP's text encoder, optimizing language prompt *within the model's representation space*. In other words, VR and prompt tuning work at different levels.\n\n- Given this difference in mechanism and locus of adaptation (input images vs. internal text embeddings), we consider them to address *different research questions* regarding model adaptation, which is why a direct comparison was not included in the initial submission.\n\nWe will clarify this in the related works of the revision.\n\n\n**W4:** Regrading the best performance and external models\n\n**R4:** Thank you for your concern. Although DVP-cse generally outperforms DVP-cls, the standalone DVP-cls still shows significant advantages over the baseline DVP. For example, it leads by 3%, 2.5%, and 2.1% on UCF101, Cars, and Aircraft, respectively. This demonstrates that the proposed DVP-PRM framework still holds clear advantages without external models.\n\n---\nThank you again for your time!", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper builds upon prior research in model reprogramming and visual prompting on CLIP. Unlike traditional VR methods that use a single visual prompt, this work introduces Decoupled Visual Prompts (DVP), which enhances learning capacity and interpretability by optimizing multiple prompts separately based on explicit causes (DVP-cse) or unsupervised clusters (DVP-cls). This approach advances the field by addressing the limitations of prior methods (VP, AR, AttrVR) and integrating a probabilistic reweighting matrix (PRM) to refine prompt contributions. The insights gained from this work may impact future research in explainable AI, and self-supervised adaptation of large-scale vision-language models. \n\nHowever, its applicability to other VLMs or generative tasks and larger datasets remains an open question for future exploration.", "essential_references_not_discussed": "NaN", "other_strengths_and_weaknesses": "Strength:\n- The paper introduces decoupled visual prompts, which optimizes multiple prompts separately using explicit causes (DVP-cse) or unsupervised clusters (DVP-cls), overcoming the limitations of single-prompt methods, provide a bit more explanability of the CLIP.\n\n- Theoretical analysis proves that DVP lowers empirical risk, and extensive experiments demonstrate consistent performance improvements over existing visual reprogramming methods.\n\n- The probabilistic reweighting matrix enhances interpretability, providing insights into how different visual prompts contribute to classification decisions, with visualization support.\n\nWeakness:\n- The method is limited to classification tasks on CLIP, and its applicability to other VLMs such as SigLIP or other vision tasks like object detection or segmentation is not explored.\n\n- Training complexity increases due to the need for optimizing multiple visual prompts, and the probabilistic reweighting matrix may not scale well in low-data regimes.\n\n- The paper does not compare against prompt tuning methods like CoOp or CoCoOp.\n\n- The best perofrmance always shown in the experiments of the DVP-cse variance, making the model rely more on external model.", "other_comments_or_suggestions": "See above comments.", "questions_for_authors": "See above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a Decoupled Visual Prompts algorithm with reweighting strategy, to improve the VR method, as an alternative of existing PET methods.   The authors also find the paper well-structured and the proposed algorithm is supported by the experiments.  There are some discrepancy during the rebuttal, and the authors address them well. Please include all of the additional results and clarification for the final version.
Decision: Accept (poster)