Bayesian-guided Label Mapping for Visual Reprogramming
Authors: Chengyi Cai
Zesheng Ye
Lei Feng
Jianzhong Qi
Feng Liu
Venue: NeurIPS 2024 oral (2024)
Keywords: visual reprogramming
adversarial reprogramming
output label mapping
Bayesian probability
Primary area: safety_in_machine_learning

Abstract: *Visual reprogramming* (VR) leverages the intrinsic capabilities of pretrained vision models by adapting their input or output interfaces to solve downstream tasks whose labels (i.e., downstream labels) might be totally different from the labels associated with the pretrained models (i.e., pretrained labels). 
When adapting the output interface, label mapping methods transform the pretrained labels to downstream labels by establishing a gradient-free one-to-one correspondence between the two sets of labels.
However, in this paper, we reveal that one-to-one mappings may overlook the complex relationship between pretrained and downstream labels. Motivated by this observation, we propose a ***B**ayesian-guided **L**abel **M**apping* (BLM) method. 
BLM constructs an iteratively-updated probabilistic label mapping matrix, with each element quantifying a pairwise relationship between pretrained and downstream labels.
The assignment of values to the constructed matrix is guided by Bayesian conditional probability, considering the joint distribution of the downstream labels and the labels predicted by the pretrained model on downstream samples. Experiments conducted on both pretrained vision models (e.g., ResNeXt) and vision-language models (e.g., CLIP) demonstrate the superior performance of BLM over existing label mapping methods. The success of BLM also offers a probabilistic lens through which to understand and analyze the effectiveness of VR.
Our code is available at https://github.com/tmlr-group/BayesianLM.

Bayesian-guided Label Mapping for Visual Reprogramming
Chengyi Cai
Zesheng Ye' Lei Feng² Jianzhong Qi' Feng Liu
'The University of Melbourne SSinaapore University of Technology and Design {chengyi. cail, zesheng. ji,jjhannng qi) qin edu au feng_ lei@sudd sgu.sg fengl wtllluumlllpp. ml@gmail. com
Abstract
Visual reprogramming (VR) leverages the intrinsic capabilities of pretrained vision models by adapting their input or output interfaces solve downstream tasks whose labels (i.c., downstream labels) might be totally different from the labels associated with the pretrained models (i.e., pretrained labels). When adapting the output interface, label mapping methods transform the pretrained labels to downstream labels by establishing gradient- one-to-one correspondence between the two sets of labels. However, in this paper, we reveal that one-to-on to-one mappings may overlook the complex relationship between pretrained and downstream labels. Motivated by this observation, we propose a Bayesian-guice guided Label Mapping (BLM) method. BLM constructs an itrrativelyupddated probabilistic label mapping matrix, with each element quantifying pairwise relationship between pretrained and downstream labels. The assignment values to the constructed matrix is guided by Bayesian conditional probability, considering the joint distribution of the downstream labels and the labels predicted by the pretrained model on downstream samples. Experiments conducted on both pretrained vision models (e.g.. ResNeXt) and lasion-language models (c.g., CLIP) demonstrate the superior performance of BLM over existing label mapping methods. The success of BLM also offers probabilistic lens through which to understand and analyze the effectiveness of VR. Our code is available at https: //g/ tigb.bb........ com/tmlr group group/Bayes Bayess ianLM.
Introduction
Repurposing pretrained models from data-rich domains [6, 28, 59] has emerged as effective strategy to address downstream tasks without re-training a task-specific model. For visual tasks, visual reprogramming (VR) [3, 4, 49, 52]-also called adversarial reprogramming [12, 38, 48] repurposes a pretrained model for downstream tasks without changing the model. In particular, VR (full task setup detailed in Appendix A) modifies the model's input interface by adding trainable noise patterns to the images of downstream tasks. Since pretrained and downstream tasks typically have distinct label spaces, label mapping (LM) function is needed to map outputs of the pretrained models to downstream labels. Often, existing VR methods adopt a gradient- one-to-on mo-eto-one LM [4, 12, 48), avoiding the computational cost of training fully-connected output layers through backpropagation.
However, we find that pretrained and downstream labels, which may limit the performance of VR. In Figure 1, we repurpose a model pretrained on ImageNet [46] for downstream classification tasks using a one-to-one LM strategy [4], and present statistical results. The two subfigures Figure la and Figure 1b, illustrate drawbacke from the perspectives of individual images and the entire dataset, respectively. Figure la shows the distribution of logits (i.e., model output before the softmax layer) for the most likely
LM overlooks the complex many- -t-y--t--ee to-many relationship between
*Correspondence to Feng Liu manggli.mllggaai...
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Drawbocks Over Individual Images
(b) Drawbacks Over the Entire Downstrean Datasse
Downstream Label >
Pretrained Model Outout
Optimal d" Mapping)
Protriine label
Pracdonta
CITRRI Dog
kanalean
Aguanra quiid
Dowsstream Label
Prezzzined Outut
Optimall /" (1-to-l Mappingi
Fai spiasi
230
morag ne 317
23
2004
Flowers
CIFARIO Devenstican Label
Osizeeprmam
Optinal to Mapping
Iguored PrebabiliGes
Suboptimal Solution
1 -P-x")
Figure I: Drawbacks of one-to-one LM from the perspectives of (a) individual images and (b) the entire dataset. An ImageNet-pretrainee classifier is reused in downstream tasks. In (a), images 'Dog" and "Osteospermum" from downstream tasks are mapped into only one pretrained label, respectively, ignoring other probabilities. In (b), the distribution of [predicted pretrained label y², ground -trtthh downstream label yy pairs reveals the existence of suboptimal solutions, where 'Automobile' cannot be paired with the optimal pretrained label 'Moving Van', which has already been mapped to "Truck".
predicted pretrained labels of two images from downstream tasks: a 'Dog' image from CIFARIO [30] and an 'Osteospermum' image from Flowers 102 [41]. For the 'Dog' image, multiple pretrained labels -subllases of dogs-receive high logits. Similarly, for the 'Osteospermum'
like 'Chihuahua', image, pretrained labels such as 'Sea Urchin', 'Daisy', which share similar features, also score high. Despite these connections, the one-to-one LM retains only the label with the highest logit, suggesting the probabilities of other related labels are ignored. Figure lb shows the frequency distribution of the predicted pretrained labels and the ground-truth trouh downstream labels of downstream samples, with the diagonal representing the results derived from one-to-one LMM The 'Automobile' class from CIFAR 10, for example, can no longer be paired with the optimal pretrained label 'Moving Van', which has already been greedily mapped to the label "Truck,, implying suboptimal label assignments.
The above observation motivates to go beyond these binary mappings. In Section 3, we replace the one-to-one LM function with a probabilistic LM matrix. Each matrix element is a real number that quantifies the relationship between pretrained label and downstream label, updated iteratively during VR optimization. This allows predictions for each downstream sample to consider diverse contributions from all pretrained labels, enabling a flexible many-to-many mapping strategy.
Specifically, we present Bayesian- guiiedd- label mapping (BLM) in Section 4, which assigns values to elements in the probabilistic LM matrix based on Bayesian conditional probabilities, derived from the joint distribution of the predicted pretrained labels on downstream tasks and the ground- truth downstream labels. We further extend BLM to BLM+, which aggregates top-K predicted probabilities instead of using a single predicted label when estimating the joint distribution, accounting for uncertainty in the predictions. We also provide a theoretical analysis that justifies the potential of probabilistic many-to-many LM to outperform deterministic one-to-one LM.
To show the effectiveness of BLM, experiments are conducted on 12 widely used datasets, with BLM and BLM+ being applied to different input VR methods methods-padding and watermarking- on pretrained ResNet and ResNeXt (see Section 5). The ablation study and parameter analysis are also included, along with visualization results and discussions of why VR is effective. BLM and BLM+ are also applied to vision- vision-language models (see Appendix L.) to demonstrate their general applicability.
In summary, both theoretical analysis and empirical findings (Tables 1-2) provide compelling evidence that BLM and BLM+, grounded in Bayesian principles, facilitate VR to leverage pretrained knowledge for diverse downstream tasks. Beyond performance improvement, BLM and BLM+ offer insights into understanding the effectiveness of VR (Figures 3-4): revealing the relations between pretrained and downstream label spaces may guide future studies into more interpretable VR methods.
2 Related Works
Model Reprogramming. Among cutting-edge transfer learning methods (see Appendix B), model reprogramming introduces an efficient learning framework for adapting models pretrained on large-
2

---

scale data to downstream tasks constrained by limited resources [7]. By changing the input or output interfaces (i.e.. input or output space) purposefully, while preserving the integrity of the pretrained model, knowledge can be reused on new tasks, sidestepping exhaustive finetuning of the model.
Many recent studies focus on repurposing diverse pretrained models for downstream tasks, including pretrained vision models [1, 4, 38, 48, 49, 52] such as ResNet [17] and ViT [11], language models [15, 50] such as BERT [24]] acoustic [21, 60, 61] and graph models [23]. Such repurposing encompasses several types: cross-modal (e.g., from voice to time-series [61]. or vision to text [38J), different tasks within the same modality (e.g., from image classification to cut-of-distribution detection [52]), and different domains within the same task (e.g., from ImageNet to medical images [48]).
Prompting and Input VR. Prompting incorporates meticulously designed prompts (additional param- eters) into pretrained models with specific architectures to utilize pretrained models in downstream tasks. Leveraging ViT, VPT [22] integrates prompts alongside image embeddings, while EEVPT [16] further enhances VPT by embedding parameters within self-attention layers. TransHP [53] additionally learns prompt tokens to encode coarse image categories. In vision-language models such as CLIP [45], besides rext-prompting methods such as CoOP [68] and CoCoOP [67]. models like MaPLe [25] also learn layer-specific mapping functions that bridge vision and text.
Slightly different from prompt tuning, input VR offers a model-agnostic approach by introducing trainable noise to images in the input space before feeding those images into pretrained models. This process does not impact the visual effect of the images. Two prevalent techniques are padding- based VR and senermaariing-sseedd VR. Padding-based based models [4, 12, 48, 49] preserve the integrity of images while introducing trainable noise patterns to the outer frames around images, whereas watermarking-basee based models [1, 3, 42, 52] train noise patterns that overlay the images.
Output Mapping for VR. Because pretrained labels and downstream labels are often different, relying solely on input VR may be insufficient for downstream tasks. To bridge this gap, output mapping methods are introduced to facilitate alignment between different label spaces. Mainstream approaches include deep learning- based and statistical inference-based (i.e., gradient-freed LM methods. Deep learning-ban based methods insert a learnable fully connected layer to connect pretrained and downstream labels [27, 49]. However, for tasks with large label spaces, the additional model layers would result in extra training costs, potentially canceling the efficiency advantages of VR.
As for gradient- free LM methods, random label mapping (RLM) [12] establishes mappings between an equal number of randomly selected pretrained labels and downstream labels, masking out other unused ones. Frequent label mapping (FLM) [48] selects optimal one-to-one mappings using greedy approach based on the number of pairs between pretrained and downstream labels. Iterative label mapping (ILM) [4] extends FLM by updating mappings at each epoch, refining the output label mapping as input VR patterns evolve. As depicted in Figure 1, these one-to-one mappings overlook potential probabilities and lead to suboptimal solutions. We propose BLM to address these issues.
3 Problem Formulation
Problem Setup. Consider pretrained task with input and output variables X and Ys, jointly defined over X x y where X C Rdds has the input dimensionality ds and yos {1,.... ks). We have pretrained classifier fpre X + Rs producing a logits vector fpre(z E R for each x E XS5 For a downstream task with input and output variables XT and YT defined over X' x y,, where X Rdr has the input dimensionality dr and y' = {I..... ). VR seeks to adapt fpre the downstream task without modifying its parameters. To achieve this, VR introduces two functions: 1) input VR function (-00) X + 2 with learnable parameters @ that converts downstream inputs for compatibility with fpee: and 2) output LM function (:) Rs + Rr that aligns the output logits of fre with the downstream label space by a transformation w. Concretely, given a training dataset DT {I, y ) with n training samples drawn from XT x x for the downstream task, the training objective of VR can be formulated as:
min GE T2 Σ ((yTT (Jor fpre o (n))aF:)))
where E is loss function, and SoL o "pre Jin denotes the composition of input VR, pretrained model and output L.M. In this study, we focus on gradient-free LM, where fou does not introduce additional trainable parameters but strategically leverages fin and fire to determine w.
3

---

Modeling Existing LM. As mentioned, fout serves to find a mapping between each y² E y and v" E y". This can be achieved by constructing an output label transformation w such that for each downstream sample x?. its label oن is predicted by arg max softmax with:
W1,1
wr
9:
=
S(TT) - [)(xT)
(2)
Wkg.11
where f(a]) is shorthand for (Spre o (x,
(x:T;) 8). w can be updated iteratively [4] with input VR. A deterministic one-to-one relation between y and yr implies only a single "correct" y y exists for each y" E y". Formally, w in Eq. (2) is binary matrix, where just a single element wj.k is set to 1 in each column of (i.e., w {0, 1) ja satisfying - 1).
Our Probabilistic LM. Considering aforementioned drawbacks of me-to-one mappings, we propose a probabilistic LM for VR, assigning real values to all elements in w (i.e., w E [0, ]] satisfying Iks
Lj=1 wy, = 1). Each element wy""," quantifies the relationship between S² E y> and y" E y This acknowledges contributions from all pretrained labels for the prediction of downstream samples. The flexible many-to-many LM implies the inherent complexity in label correspondence. In Section
4, we investigate how to assign values to our probabilistic LM based on Bayes' theorem.
Bayesian-guided Probabilistic Label Mapping (BLM)
4.1 Method Demonstration
Interpreting ( The objective of VR is to maximize p( p(YT|XT) defined over the down- stream task space. By using the law of total probability, we can express p p(YT|XT) as
p(Y "Ix") E P(Y p(Y x1(x²) p0 "Ir =,,," x*"
(3)
Mirroring the structure of Eq. (2), Eq. (3) enables us to estimate p using training data D
7
p(y - - x p(Y = Me 8
xT 2
(4)
@ input VR:
outpat J.M
where ① denotes the predicted probability of pretrained label y for input T._ obtained from fpre o fin- Essentially, I can be viewed the standard input VR and is orthogonal to the LM methods employed; @ represents the probability that the true downstream label y;" is mapped from the predicted y² and input x!", which amounts to estimating the output label transformation w E [o, 1| independent of output L.M, the focus now shifts to estimating @.
Since ① is
Estimating Sy Using Conditional Probability. Since wiy . is used to quantify the contributions from pretrained label y to downstream label y we can associate with the conditional probability:
D T 3. x p(Y = T. Y y
(5)
p 1=|
By applying Spre fin to DT we can empirically estimate the joint distribution of p(Y = ys (x)), then obtain p(y' = yrIXT) EyJeyr p(YT = = y' = yf and substitute them into Eq. (5). Two strategies, BLM and BLM+, are presented for these estimations in this paper. To help understanding, we include a simple example to illustrate the estimation of p(YT = y", Y x and p(Y'S = -5(XXT in Appendix C.
BLM. Let f(x+) = (Spre {in) model for given input ..f. We define of arg eys f(xx' yy to be the predicted pretrained
; 0) denote the predicted logits obtained from the pretrained
label for xif and I to be the indicator function. Starting with the joint distribution p(y'T
y Y =ySh
= XT), we could intuitively count the frequency of (0. S A vI =y") to estimate:
P =T ys 11ITT)= En 1l )
(6)
"This estimation is similar to [40], see Appendix for more discussion.

---

Input Visual
Bayesian-guided Mapping
Reprugramming
Inference
Stotiaties
Fixedl Pretruined Model
Backpropagation
Pretrained Labd
Step 2
Probabilistic LM
Downstream
Padding
Step 1
Legis
Prequency Diatibont
Predieted
Task
OR
Step
Downstram Labe
Grosd Trush
Predicted
Imput Image
Protesined Label
Step 4
Watermarkinn
Figure 2: Learning strategy of BLM and BLM+. First, input images, incorporated with VR water- marking or padding patterns, fed into fixed pretrained model to obtain logits and predicted labels. Then, the true labels (of y") and predicted labels (of y²) are used to estimate WBLM or WBL. Next, using WBLM WBLM, that reweights output logits of pretrained models for the downstream labels, the predicted results can be derived. Finally, backpropagation is performed to update the input VR.
For p(Y' y IXT), in addition summing up Eq. (6) for yJ E yT, add Laplace smoothing coefficient À to ensure the denominator of Eq. (5) being non-zero, with ks being the size of y
PBLM(Y = - EyTeyr EE= 1/ = v 1 = + E 2- 1 v +
(7)
ks
n+ ks À
Substituting Eq. (7) and Eq. (6) back to Eq. (5) yields the estimation of w3y "T be PALM(Y
kts J , XT). After column-wise wwie sum normalization of ", to satisfy 2j=1"1 = 1 (as
formulated in Section 3), we obtain the final probabilistic LM, denoted as WALM.
BLM+. Recall that BLM estimates p(Y ,,, Y -r|XT) by trequency-counting based single most likely predicted label. However, this strategy disregards other high-ranking predictions that could offer valuable information. Thus, introduce BLM+, extension of BLM that considers top- K predicted probabilities of the pretrained model for the estimation of p(Y-T = y", y = )). Rather than relying solely the tally, BLM+ aggregates probabilities for samples where y ranks among the top-K predictions. In this way, BLM+ acknowledges the uncertainty in fxx²) and exploits other potential predictions, providing more robust estimations.
Let D = {1/l arg ma....... J (x )r' denote set of the top-K predicted pretrained labels for input T and ((uS)='''> (softmax (xx y denote the predicted probability for any yو y) given n²f" Then, within the BLM+ strategy, the joint density is approximated as:
> 1l y²} p(y1kT)) 11 E v
PBLM, (YT =-==y" =y =²1x
(8)
Similar to BLM, with the Laplace smoothing coefficient being À and the size of ys being ks. p(Y -y=s can be expressed by applying BLM+ as:
ÍBLM (Y - Σ ply af) 1 E DR +1
(9)
n 1.
Combining Eq. (9) and Eq. (8) with Eq. (5), and going through all y yT and yd y,S, we obtain the full BLM+ estimation as WBLM,, after column-wise sum normalization of م similar to BLM. In practice, we set K a hr J. with ratio being a hyper- parameter that decides K based on the size of downstream label space kr-
Pipeline and Learning Strategy. The learning of BLM and BLM+ allows for seamless integration into existing VR pipelines. It model-agnostic (e.g., pretrained ResNet ResNeXt) and compatible with all input VR methods (e.g., watermarking or padding). Figure illustrates the learning strategy in detail. Besides, the learning pipeline of BLM is shown Algorithm while that of BLM+ shown in Algorithm 2. The completed pseudocode for all LM methods (RLM, FLM, ILM, BLM, BLM+) and more detailed discussion of involved matrix operations are in Appendix D.
Note that this approximation not normalized, and thus, is not strictly equivalent to the probability.
5

---

Algorithm 1 Training Pipeline of BLM
Algorithm 2 Training Pipeline of BLM+
1: Input:
Pretrained label space ) with ks la-
1: Input: Pretrained label space ys with ks la- bels, downstream label space y with ky labels, downstrcam training set {[=:] V/ TT M pretrained model f-re(),, iterations E, learning rate X, K
bels, downstream label space y" with fr la- bels, downstream training set {xx V/
(/))) )== pre-
trained model f-re()), iterations E, learning rate a, hyper-parameter À
2: Output: Probabilistic LM WBLMM E [0, 1
2: Output: Probabilistic LM WBLM E [0, xer
x
3: Initialize WBLM+ < {0) xa set 0
3: Initialize WBLM < (0)& set 4- 0
4: for = 1...E do
4: for e. = 1...E do
5: # Step I: Get Pretrained Model Outputs
5: # Step 1: Get Pretrained Model Outputs
6: f(=];; :0) fpre ; 01) for 1...
6: [(x:"::) fpro(fi -))
I...n
7:
# *Seep 2: Compute (or Update) the LM Matrix
7: Step 2: Compute (or Update) the LM Matrix
8:
>i., < (y/largmxx,,... p1...." [(x 0)y/ for 1...
8:
s/² < argmax ye [=];;), for = I...n
9:
p(ul:?) +- solimax(f(x; ())))()))))])))))))))))))))))))) for y = 1...
9:
if then Compute WBLM using Eq. (5,6,7)
10:
if e=1 then Compute WBLM+ using Eq. (5,8,9)
10:
else Update WBLM using Eq. (5,6,)
else Update WBLM+ using Eq. (5,8,9)
11:
Step 3: Predict Downstream Labels
12:
Step 3: Predict Downstrcam Labels
12: 9." + argmax, faur (a. ( 0)1, for 1...
13:
o. +- argmax, ( (f(2.) ()), for 1...n
13: Step 4: Update VR Patterns
14: Step 4: Update VR Patterns
14: a 0 Ve E flT foa ((()):0)))
15:
Vo Eis A(u Farr (f()):0)))
15:
end for
16:
end for
16:
return WRLM
17:
return WBLM+
The iterative process of learning WBLM, WBLM+ comprises these four steps: 1) Input images, with VR patterns, are fed into the fixed pretrained model to obtain output logits and predicted pretrained labels. 2) BLM and BL.M+ replace previous L.M (e.g., RLM, FI.M or IL.M) to estimate w. 3) The initial logits are reweighted using WBLM or ,BLM, yielding refined predictions for downstream labels. 4) Loss functions (e.g., cross-entropy) and backpropagation are employed to update the input VR.
4.2 Theoretical Analysis
Furthermore, we include a justification of why probabilistic many-to-many LM (e.g.. BLM and BLM+) should be favored over deterministic one-to-one LM (e.g., RLM, FLM and ILM). Define the label spaces y fo. 1} and yT f0. 1} as binary sets". Consider the set of potential LM functions Fim [fim y + y"" including each function (1²) E {y y²} For any fim E Fim. the expected accuracy of fim regarding the entire downstream label space is defined as":
Acc(fim) = ESTEyT
Σ p(u-) (fmolv²) = ( =²11²) SEys
(10)
where p(yS) is the marginal distribution of the pretrained labels and p (fim(yS) = y v/'lsS) is the conditional probability that fim correctly predicts a downstream label y from a pretrained label y Let fplm and falm denote the probabilistic LM (Definition E.1) and deterministic LM (Definition E.2),
respectively. We finally prove that (cc(f 2 Acc(faml (Corollary E.5) in Appendix E, which further verifies the effectiveness of our methods in the view of theoretical understanding.
Experiments
Tasks and Baselines. Following ILM [4], we employ ResNet- 18 [17] pretrained on ImageNet-
[46] and ResNeXt pretrained on Instagram [37] to test the performance of VR. The results are
evaluated on twelve downstream datasets: Flowers 102 [41], DTD [9], UCF101 [47], Food101 [2],
GTSRB [19], EuroSAT [18], OxfordPets [44], StanfordCars [29]. SUN397 [58], CIFAR 10/100 [30]
and SVHN [39]. Previous gradient- free LM methods RLM [12], FLM [48] and ILM [4] are used
as the baselines. The results of deep learning- based LM will also be included for reference, where
LM is treated as a single-layer linear neural network connected to the output of the pretrained model
"This analysis focuses on the binary setting for simplicity.
Input is intentionally omitted as all LM methods operate the same inputs.

---

Table 1: Performance comparison of gradient- freee output LM methods (mean std %). Ours are highlighted and the highest accuracy is in bold (with deep learning-baat baaed LM in gray for reference)
ResNett-18 (ImageNet- IK)
ResNeXt- 101-1000-32xx0 (Instagram)
Padding
Gradient-free
Deep
Grudient-
Deep
Methods
RLM
FLM
IIM
BLM
BL+++
FLM
IIM
BLM
ML
100
11.0as
20,0.a
27.9wm:
44.4
50.1.3
76.7wa
22.5
27.9mm
31.5ma
30.1-a>
85 2512
DTD
16.3
32.4us
35.3we
42.0uss
43,9.24
49.1uns
40.3
41.4wa
47.8
49.4.a4
64.0uz
UCF101
6.6aa
18.9
23.9as
30.9Au
32.0.a4
46.0as
41.9at
43, Los
48.3
50.
68.3
00od101
3,8
12.8
14.8
23.2pM
25,
34.1au
20.5us:
23.0ws
2u,6
31.4.a:
GTSRB
46, د
45.
52.0at
54.8wz
54.3ua7
63. Lans
56.2ass
59.9ain
62. 9uss
63.0.as
74, 74.4us
Eu
82.4
83.8
85.2
86.7pz
86.7.11
92.4mu
87,8
86. 2mxx
87, 6uss
88.3.a.
93.2au
OxfordPets
9.3mma
62.9uma
65.4ux
69.8uss
70.6.a1
73.0us
76.8
78.9wx
82.4
83.0.as
91. Rar
StanfordCars
0.9mu
2.7
4,5
5,4
7.7
14.3
4.6m1
7,0
8.3
9.33ms
50.5
SUN397
1.0au
10,4
13.0mm
16.2au
18.7:1s
26.3mm
21.6un:
23.7wz
30. Iau
32.0.a.
51 51.5aus
CIFAR1O
65.7-ex
65.5.e1
66. 6617
66.8.1>
72.1ws
80.3w
81,7ws
82.2
82.2-a
83.4
CIFAR100
12.9
18.1
24.8.a1
29. 6mn
30.6-14
46.7am
39.7.a3
45.9wm
47.8
47.8
56.2ma
SVHN
73.5w33
73.1
75.2mm:
74.5wo
74.2-3>
82.
79.0ws
81,4
79.8wa
79.3
85.7wa
Average
27.2
37.2
40.6
45.3
46.7
56.3
47.6
50.0
53.2
53.8
71.9
for training alongside VR. More dataset and implementation details are in Appendix F. Regarding hyper-parameters of BLM, À set as and the top-K ratio a is 0.15 (analyzed in Appendix G).
Results for Padding-based VR. Padding- based input VR adds trainable noise to the outer frames of centered images. Table shows the performance of BLM and BLM+ applied with padding- based input VR. BLM and BLM+ yield the highest accuracy across all datasets except for SVHN. On ResNet-18. 18, compared to the SOTA (i.e., ILM), BLM achieves an average improvement of 4.7% across the 12 datasets, whereas BLM+ achieves a 6. % enhancement. On 1e, BLM and BLM+ achieve accuracy improvements of 3.2% and 3.8% on average, respectively. The elevation in accuracy is particularly pronounced in tasks with a higher number of classes (e.g.. UCF101, CIFAR 100). On SVHN, ILM performs slightly better, which could be attributed the minimal inter-class variation and the smaller number of classes (which is 10) in SVHN, resulting in similar mapping values for different downstream labels and thus reducing our method's s advantage (discussed in Appendix H). However, compared to current gradient-free LM methods, the deep dearning-basee LM may still have an advantage in the performance of downstream tasks due to the learning capacity of the linear layer neural network. Our proposed BLM and BLM+ aim to bridge the gap between gradient-fres free LM and deep learning-basse interpretabilityy (see Appendix for more experiments) and fewer parameters (see Appendix J for details) compared to deep learning-based LMM
LM. Additionally, BLM and BLM+ have been observed to possess greater
Results for Watermarking-based VR. BLM and BLM+ can be applied to different input VR methods. For the watermarking- based VR method, which overlays trainable noise pat- terns on resized images, the results of BLM and BLM+ with ResNet-18 as the pretrained model are shown in Table 2. Since ILM is the best-performing baseline, we only include its results here for comparison. Our BLM and
Table 2: Performance comparison of gradient- VR
free LM methods for watermarking- on R8 (mean % std %). Ours are highlighted and the highest accuracy is in bold with deep learning-basee based LM in gray for reference)
Watermarking
Grudient-free
Deep
Methods
ILM
BLM
BLM+
Flowers102
23.2ms
39.2ma
44.
8244a
BLM+ methods again outperform ILM, achiev- ing an average gain in accuracy of 6. 6.1% %% and 7.5%, respectively. Therefore, in the case of natermarking-basee VR, BLM and BLM+ also close the gap between current gradient- free and
DTD
29.0wm
40.
43.0aaa
48.9.ss
UCF101
24.4mm
32.9as
35.4
53.
Food101
13.2
2l.5
22.9ka
30.4as
GTSRB
76.8
82.
82.0.x
89.5as
AT
84.3
84.4was
84.8.aa
89.2as
deep learning-base LM. Results in Tabel 2 underscore the applicability of our output LM methods with different input VR.
OxfordPets
70.0aa
72.4-am
73.3
77.6ax
StanfordCars
3,4
5.5-o1
7.4..
30.7-aa
SUN397
13.4mz
18.4am1
19.4.o2
32.9au
CIFARIO
68.9wm
74.9ua
75.7.a1
71.7ms
CIFARI00
33.8
41.2-a3
41.6Go
39.9as
Results for Hision-Language Models. The
SVHN
78.3
79.2-k1
78.8
83.7-m
application of our BLM and BLM++ on vision-
language models (i.e., CLIP), along with the performance, and visualization results are dis-
Average
43.2
49,3
50.7
60.8
7

---

Dowasiream Task
Top Weighted Prefrained Labels Applying BLM++
Top W Weighted Pretrained Applying BLM
Or
0.0907 X Green stake
0L0828 X Articloke
0.0803 X
0.1010 X Green mambe
0.0862 x Green stake
G. 0,000 x
Edamname
Greas mamba
Vine stuke
X
0.0693
1,0508
0.0879
Fibrous
Porcupinc
Porcupine
CIFARIO
0.0386
0.0381
0.0367
0,0320
Cocker Spanic!
Fuglish
Eaplish springer
Wire-haired
Figure 3: Visualization results of top weighted pretrained labels y and weights wys.y" for some y² applying BLM and BLM+. Downstream labels 'Edamame', 'Fibrous', and 'Dog' are shown as examples. ResNet- pretrained ImageNet is used. More results are in Appendix K.
cussed in Appendix L.. BLM and BLM+ achieve the average accuracy of 79.1 % and 79.3% across 12 datasets, respectively, and outperform the baseline methods on 11 datasets.
Ablation Study. Table 3 presents the ablation study results BLM and BLM+. For BLM, we list the results of replacing probabilistic LM with one-to-one LM, denoted as '-Bayes', and the results of calculating WBLM only once in the first epoch without subsequent iterations, denoted as '-Iter'. For BLM+, the removal of aggregating probabilities results in BLM; hence, report the results of aggregating all probabilities instead of top- predicted probabilities, denoted as '-Top-K'. Like that for BLM, 'Iter' shows the results calculating WBLM+ only once without subsequent iterations. Besides, when both -Top-K' and -Bayes' are applied to BLM+, BLM+ degenerates into the same results as 'Baayss BLM, which is displayed in the previous column of Table 3.
For BLM, employing 'Bayes' compute improves curacy across most datasets, with slightly smaller gains ob- served for datasets with fewer classes (EuroSAT, CIFARIO, and SVHN). For BLM+, ap- plying 'Top-K' assists filter- ing out redundant information, yielding positive impacts across
Table 3: Ablation study results of BLM and BLM+, using ResNet- 18 as the pretrained model (showing the mean accuracies (%), with ours highlighted and the best in bold)
Method
BLM
BLM+
Bayes
Iter
Ours
Top-K
Iter
Ours
Flowers102
27.9
30.8
44.4
48.2
43.6
50.1
DTD
35.3
38.0
42.0
42.4
42.0
43.9
UCF101
23.9
28.8
30.9
30.9
33.2
32.0
most datasets. In particular, on the OxfordPets dataset, to classify cat and dog breeds, us- ing top-K predicted probabil- ity effectively filters out numer- ous irrelevant categories in the pretrained label space, which leads to significant improve- ments. Furthermore, for both BLM and BLM+, iterative up- dates crucial as the initial input VR may deviate considerably from the final iteration. The greater the disparity between the domains of downstream and pretrained tasks (GTSRB, SVHN), the more pronounced the impact of the input VR, thereby emphasizing the necessity of iteration updates.
Food 101
14.8
18.4
23.2
23.6
22.8
25.1
GTSRB
52.0
44.6
54,8
50.5
44.8
54.3
EuroSAT
85.2
85.0
86.7
85.1
85.4
86.7
OxfordPets
65.4
68.1
69.8
59.9
70.6
70.6
StanfordCars
4.5
2.8
5.4
6.5
6.2
7.7
SUN397
13.0
14.5
16.2
17.3
16.4
18.7
CIFARIO
65.5
63.9
66.7
65.2
63.5
66.8
CIFAR100
24.8
23.2
29.6
30.8
26.2
30.6
SVHN
75.2
64.6
74.5
70.0
62.6
74.2
Visualization Results. The probabilistic LM obtained by BLM or BLM+ can elucidate the connection between pretrained and downstream label spaces. Figure shows the visualization results for three

---

Task:
Reprogramming
102
Marigold
Vieual
0.151
X
0.347
0.179 X
0.367 X
DJIKS x
0,194 X
0.385 x
0.184 x
0.183
0.1K3 X
Soral Fee
Saniiflower
Sauliflower
Canlifluwer
caulsfowerr
Cantiflower Couliflawer
Cauliffower
cauliflower
cuuldfoower
Top Weeegeee Premnined
Labals Applyinggliiiiie.ppeeepppppppppp BLM+
10.34
+0.344
0.134
neckBront
0.066
0.0054
0.0009
0.074
0.004
0.075
SIDCCOOL
pincupple
pincappie
Weight Changes between Adjucent Epochs
2.5
Training Loss
2.0
Epochs
20
40
60 80
100
120
140
160
180
Figure 4: Visualization of input VR and top-weighted pretrained labels applying BLM+. Training loss and weight changes (Euclidean norm) of probabilistic LM WBLM+ per iteration are plotted below. Pretrained ResNet-18 18 is used, and the downstream label 'Marigold" is selected example.
example. Each column of w computed using BLM or BLM+ is a vector with length kg = 1000, representing the weights assigned to the 1,000 outputs-one for each -o the pretrained model corresponding to a downstream label y². The top-weighted labels (i.e., n where Wy*.aT is larger) for 'Edamame' correspond to organisms such as snakes and artichokes, which share similarities in color and shape. Similarly, the predominant labels associated with 'Fibrous' from the texture dataset include rough- textured items like 'Hay" and 'Komondor'. 'Dog' encompasses various sub-breed canines. These findings suggest that BLM and BLM+ establish an optimal probabilistic LM between label spaces, and handle similarity or inclusion relationship, addressing the drawbacks in Figure 1.
labels in downstream tasks, taking ResNet-18 18 pretrained ImageNet-1
Discussion of Why VR Is Effective. From a visualization perspective, Figure shows the top- weighted pretrained labels and input VR patterns f at different iteration stages using BLM+. The training loss for each iteration and changes in W, measured by the Euclidean norm, are also plotted. During the update of wa and the pretrained labels with top wys.y" for y² being 'Marigold' transition gradually from dissimilar labels such as 'Reef" and "Teddy" to "Cauliflower' and 'Pineapple' which share more similarities in color, shape and texture. Meanwhile, the training loss diminishes gradually, and w converges, demonstrating the effectiveness of VR and BLM+.
Impact of Label Space Sizes kr- Figure 5 shows the relationship between different sizes of the downstream label space and the accuracy improvement achieved by BLM and BLM+. Tasks with larger label spaces report more pronounced performance improvements. While simpler tasks with smaller label spaces might not fully showease the power of our approach, the strength of BLM and BLM+ lies in unraveling the complex many-to- to-nayy relationship that often arises in tasks with more numerous classes. In such scenarios, our probabilistic LM methods demonstrate their full potential.
Impact of Training Dataset Sizes Figure 6 illustrates the impact of varying training dataset sizes for the downstream task on different LM methods. Regarding CIFAR as the downstream task, compared with RLM and ILM, BLM and BLM+ yield higher accuracy consistently. With approximately a 40% fraction of the downstream training data, BLM or BLM+ can achieve similar accuracy compared with training on the entire dataset.
The parameter experiments and performance analysis regarding the impact of Laplace smoothing coefficient X and top-K ratio for BLM and BLM+ are detailed in Appendix G. The visualization and analysis of LM matrices derived from gradient-free and deep learning- methods can be found in Appendix I. Training cost analysis is discussed in Appendix J. Additional visualization results of LM methods applied to pretrained vision models are presented in Appendix Lastly, the application of BLM and BLM+ for vision- iision-language models is explored in Appendix L.
Other Experiments.

---

CIFARIO
OxfordPets GTSRB
DTD CIFAR100
UCFI01
RLM
BLM (Ours)
EuroSAT
Food101
BLM+ (Ours)
BLM
BLM+
12
12
30
10.3
Accuracy Improvement (%)
10
Accuracy Improvement (%)
8.4
8.6
Accuracy (%)
25
8
8.1
6.7
20
5.8
5.2
4.4
4
15
2.8
2.3
2-
0.25
0.50
0.75
1.00
Task Data Fraction CIFARIO0
50
100
0
50
100
Sizes Downstream Label Space
Sizes Downstream Label Space
Figure 6: Accuracy (%) of methods when varying training dataset sizes n for downstream task CIFAR 100, us- ing pretrained ResNet-18.
Figure 5: Accuracy improvement (%) of BLM and BLM+ compared with ILM given different sizes (kr) of the down- stream label space, using pretrained ResNet-18.
Conclusion
focus on output LM methods for VR and reveal the drawbacks current gradient-free LM methods, which use one-to-one mappings that overly simplify the relationship between the pretrained and downstream label spaces. To address this issue, we propose BLM, which calculates probabilistic LM matrices guided by Bayes' theorem. Additionally, predicted pretrained labels instead counting a single label during the estimation of probabilistic LM matrices, yielding improved method BLM+. Both theoretical analysis and experimental results validate the effectiveness of BLM and BLM+ while offering insights into understanding the effectiveness of VR through probabilistic lens.
aggregate the probability of top-K
Acknowledgement
CYC, ZSY. and FL are supported by the Australian Research Council (ARC) with grant num- ber DE240101089, and FL is also supported by ARC with grant number DP230101544 and the &SF& Responsible Al program with grant number 2303037. JZQ is supported by ARC with grant number DP240101006. This research is also supported by The University of Melbourne's 's's Research Computing Services and the Petascale Campus Initiative. We sincerely appreciate the time and dedication of the reviewers in carefully reviewing manuscript.
References
[1] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 17274, 2022.
[2] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101-ma 1od-100--mning discriminative components with random forests. In ECCV, 2014.
[3] Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Sample-specific masks for visual reprogramming- hhase prompting. In ICMI, 2024.
[4] Aochuan Chen, Yuguang Yao, Pin- Chen, Yihua Zhang, and Sijia Liu. Understanding and improving visual prompting: A label-mapping perspective. In CVPR, 2023.
10

---

[5] Hao Chen, Ran Tao, Han Zhang, Yidong Wang, Xiang Li, Wei Ye, Jindong Wang, Guosheng Hu, and Marios Savvides. Conv-adapter: Exploring parameter efficient transfer learning for convnets. In CVPR, 2024.
[6] Hao Chen, Jindong Wang, Ankit Shah, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, and Bhiksha Raj. Understanding and mitigating the label noise in pre-training on downstream tasks. In ICLR, 2024.
[7] Pin-Yu Chen. Model reprogramming: Resource-efficiett cross-domain machine learning. In AAAI, 2024.
[8] Haoang Chi, Feng Liu, Wenjing Yang, Long Lan, Tongliang Liu, Bo Han, William Cheung, and James Kwok. Tohan: A one-step approach towards few-shot hypothesis adaptation. NeurIPS, 2021.
[9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[10] Ruijiang Dong, Feng Liu, Haoang Chi, Tongliang Liu, Mingming Gong, Gang Niu, Masashi Sugiyama, and Bo Han. Diversity-enhancing generative network for few-shot hypothesis adaptation. In ICML, 2023.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2020.
[12] Gamaleldin F Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial reprogramming of neural networks. In ICLR, 2018.
[13] Zhen Fang, Jie Lu, Feng Liu, Junyu Xuan, and Guangquan Zhang. Open set domain adaptation: Theoretical bound and algorithm. IEEE TNNLS, 2020.
[14] Zhen Fang, Jie Lu, Feng Liu, and Guangquan Zhang. Semi-supervised heterogeneous domain adaptation: Theory and algorithms. IEEE TPAMI, 2022.
[15] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. In ACL-IJCNLP, 2021.
[16] Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E vpt: An effective and efficient approach for visual prompt tuning. ICCV, 2023.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[18] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing, 2019.
[19] Sebastian Houben, Johannes Stallkamp. Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In IJCNN, 2013.
[20] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint wrXiv:2106.09683 2021.
[21] Yun-Ning Hung, Chao-Han Huck Yang, Pin-Yt Yu Chen, and Alexander Lerch. Low-resource music genre classification with cross-modal neural model reprogramming. In ICASSP, 2023.
[22] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[23] Yongcheng Jing, Chongbin Yuan, Ju, Yiding Yang, Xinchao Wang, and Dacheng Tao. Deep graph reprogramming. In CVPR, 2023.

---

[24] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. 2019.
[25] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[27] Eliska Kloberdanz, Jin Tian, and Wei Le. An improved (adversarial) reprogramming technique for neural networks. In ICANN, 2021.
[28] Jannik Kossen, Mark Collier, Basil Mustafa, Xiao Wang. Xiaohua Zhai, Lucas Beyer, Andreas Steiner, Jesse Berent, Rodolphe Jenatton, and Effrosyni Kokiopoulou. Three towers: Flexible contrastive learning with pretrained image models. NeurlPS, 2023.
[29] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, 2013.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, et al. Conditional adapters: Parameter-efficieet transfer learning with fast inference. NeurIPS, 2023.
[32] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
[33] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. NeurIPS, 2018
[34] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. NeurlPS, 2016.
[35] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: /608.009000099 2016
[36] Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmotlagh. Progressive graph learning for open-set domain adaptation. In ICML, 2020.
[37] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
(38] Paarth Neekhara, Shehzeen Hussain, Jinglong Du, Shlomo Dubnov, Farinaz Koushanfar, and Julian McAuley. Cross-modal adversarial reprogramming. In WACV, 2022.
[39] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NeurIPS workshop, 2011.
[40] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure evaluate transferability of learned representations. ICML, 2020.
[41] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing. IEEE, 2008
[42] Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In CVPR, 2023.
[43] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter- efficient tideeo transfer learning. NeurIPS, 2022.
12

---

[44] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[46] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. I/C,, 2015.
[47] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[48] Yun-Yu Yun Tsai, Pin- Chen, and Tsung- Yi Ho. Transfer learning without knowing: Reprogram- ming black-box machine learning models with scarce data and limited resources. In ICML, 2020.
[49] Hsi-Ai Tsao, Lei Hsiung, Pin- Yu Chen, Sijia Liu, and Tsung- Yi Ho. AutoVP: an automated visual prompting framework and benchmark. In ICLR, 2024.
[50] Ria Vinod, Pin-Yu Yu Chen, and Payel Das. Reprogramming language models for molecular representation learning. NeurIPS, 2020.
[51] Boyu Wang, Jorge Mendez, Mingbo Cai, and Eric Eaton. Transfer learning via minimizing the performance gap between domains. NeurIPS, 2019.
[52] Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of-distribttion detection. NeurIPS, 2022.
[53] Wenhao Wang, Yifan Sun, Wei Li, and Yi Yang. Transhp: Image classification with hierarchical prompting. NeurlPS, 2023.
[54] Zixin Wang, Yadan Luo, Zhi Chen, Sen Wang, and Zi Huang. Cal-sfda: Source-free domain- adaptive semantic segmentation with differentiable expected calibration error. In ACM MM, 2023.
[55] Wei-Hung Weng, Jonathan Deaton, Vivek Natarajan, Gamaleldin F Elsayed, and Yuan Liu. Addressing the real-world class imbalance problem dermatology. In Machine learning for health, 2020.
[56] Wikipedia contributors. Cartesian product Wikipedia, the free encyclopedia. http:://aatttttttt//eeeee 2024.
wikipedia org/w/indes: witippddaa.or/////ddee. http://teeeeeteeeeeeaappaaoriaaa
[57] Wikipedia contributors. Cosine similarity Wikipedia, the free encyclopedia. http:://ettttte 2024.
wikipedia org/v werg/w/inden /i//xxidxxxx//
[58) Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
[59] Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Fangzhou Mu, Yin Li, and Yingyu Liang. Towards few-shot adaptation of foundation models via multitask finetuning. In ICLR, 2024.
[60] Chao-Han Huck Yang, Bo Li, Yu Zhang, Nanxin Chen, Rohit Prabhavalkar, Tara N Sainath, and Trevor Strohman. From english to more languages: Parameter-efficient model reprogramming for cross-lingual speech recognition. In ICASSP, 2023.
[61] Chao-Han Huck Yang, Yun-Yun Tsai, and Pin- Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In ICML, 2021.
[62] Li Yi, Gezheng Xu, Pengcheng Xu, Jiaqi Li, Ruizhi Pu, Charles Ling, A Ian McLeod, and Boyu Wang. When source-free domain adaptation meets learning with noisy labels. arXiv preprint 230.132301 2023.
13

---

[63] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked tanguage-moeels arXiv preprint arXiv:2106.10199, 2021.
[64] Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In CVPR, 2024.
[65] Yichi Zhang, Yinpeng Dong, Siyuan Zhang, Tianzan Min, Hang Su, and Jun Zhu. Exploring the transferability of visual prompting for multimodal large language models. In CVPR, 2024.
[66] Ziyi Zhang, Weikai Chen, Hui Cheng, Zhen Li, Siyuan Li, Liang Lin, and Guanbin Li. Divide and contrast: Source-free domain adaptation via adaptive contrastive learning. NeurIPS, 2022.
[67] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
[68] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[69] Yitao Zhu, Zhenrong Shen, Zihao Zhao, Sheng Wang, Xin Wang, Xiangyu Zhao, Dinggang Shen, and Qian Wang. Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis. In IEEE International Symposium Biomedical Imaging, 2024.
14

---

A The Problem Setting of VR
Variable
Fixed
Reuse and Repurpuse
Reprogramming
Pretrained Domain
Downstream
with Abundant Data
Output Labe Mappling
Downsdrruum abeel
Pretrained Model
Preteaine Label
Pattern
Frotrained Modd
Inpur Visanl R
Pretraimed
Digits
Flowers
Figure 7: The problem setting of Visual Reprogramming. The left part shows the pretrained model and corresponding dataset, while the right part shows downstream tasks. The pretrained model is fixed, whereas the input VR and output LM modules are variable.
The task of VR reuses fixed pretrained models for downstream tasks. As illustrated in Figure 7, an input VR module operates before pretrained models, directly altering the input space of downstream tasks. Concurrently, an output LM function acts after pretrained models, taking the predicted pretrained labels input and outputring those for downstream tasks. Hence, VR achieves the reusability of pretrained models for downstream tasks without adapting the model parameters, primarily through modifications the input and output spaces.
B Recent Work in Transfer Learning
Visual reprogramming one type of methods that aim obtain models on downstream tasks with the help of pretrained models. This process is similar to the aim of transfer learning which is used leverage knowledge from data-rich domain [13] or pretrained model [51] to address tasks on other domains. The former is known as domain adaptation, and the latter is known as finetuning.
Finetuning. Given pretrained model, finetuning uses trainable parameters to accommodate specific information of the downstream tasks. As pretrained models grow in size, recent progresses in transfer learning have prioritized parameler-efficient finetuning (PEFT) [20] support resource- friendly adaptation. Regarding PEFT, the prevailing methods can be categorized as follows. The most widely adopted approach is selective finetuning [46, 63], which adjusts subset of parameters from the pretrained model while keeping the remaining components fixed, thereby reducing the total number of trainable parameters for downstream tasks. Other methods may involve adding adapters [5, 31, 43], which introduce extra trainable layers or parameters to the pretrained model and finetune only these adapters during training. Moreover, low-rank adaptation methods [20, 64, 69] have also been proposed for pretrained Vision Transformers. They apply low-rank decomposition to the parameters during training, achieving remarkable performance on downstream tasks with significantly reduced number of parameters. Additionally, Prompt Tuning methods [16, 22, 53], similarly directed pretrained Vision Transformers, integrate trainable parameters parallel to features at the input and intermediate layers. The primary distinction of these methods from VR [1, 3, 4, 48, 49, 65] lies in their necessity to be designed according to different pretrained model architectures and may also involve modifying the model weights. In contrast, VR is model-agnostic and does not require alterations to pretrained model parameters.
task-
Domain Adaptation. Domain adaptation (DA) aims to bridge the distributional gap by aligning feature spaces of the source task to the target domain with different data distributions [14, 36, 54]. Often, DA is achieved by learning invariant representations or transforming parameters to manage domain-speciffe shifts of the source and target data. CDAN [33] addresses this by introducing conditional discriminator for class label-conditioned feature adaptation, while UDA [34] leverages residual layers capture both domain domain-specific and domain-shared representations. More recently, source-free DA [32, 62, 66], which seeks adaptation without access to the source data, has gained
15

---

popularity due to growing concerns over data privacy and storage limitations, as well as the need for adaptation in scenarios where source data is inaccessible [8, 10].
Pretrained Model Selection. The BLM/BLMM+ framework models p p(YT Ix) by using the pre- the pre-traiend label space y (Eq. (3)), relying n training examples for empirical estimation (Eq. (4)). Essentially, Eq. (4) can be seen as a score that assesses how a pre-trained model, whose output lies in y performs on downstream task X x x measured over v. downstream samples. This finding resembles LEEP [40], a pre-trained model transferability metric, which shares the same empirical formulation with BLM. In this sense, BLM/BLM+ extends LEEP by incorporating top-K K aggregation for more robust estimation, suggesting the potential as more generalized framework for assessing pre-trained model transferability.
trained model Spre and integrating
C A Simple Probability Estimation Example
Ground Truth Downstream Labels y"
Downstream Task x" yr
Apply VR (x)()))) ()
Predicted Pretrained Label Di
Frequency Distribution
Cat
Dog
Cocker Spaniel
0
English Springer
0
Cat
(x,
Egyptian Cat
1
Cocker Spaniel
Joint Distribution p ",,,""
1x(" )
Pretrained Moda
Cat
Dog
Dogi
Cocker Spaniel
0
1/2
English Springer
0
English
Springer
Egyptian Cat
1/4
Dog
Distribution pVV* = xxxx
Dogi
Probability
Cocker Spaniel
1/2
Egyptian
Cat
English Springer
Cat)
Egyptian Cat
1/4
Figure 8: A simple example to help understand how to estimate p(Y v² Y 3 (T and p(Y = y I
We aim to estimate p(YT = T y= y XT) and p(Ys y XT) for BLM and BLM+ in this paper. Here, we employ a simple example (without Laplace smoothing) to help understand how to estimate these two probabilities.
The conditional probability p(YT = S = =y X represents the joint distribution of Y and Y8, given the input reprogramming f: f). the pretrained model f.re()), and the variable x of the downstream task. Similarly, p(y' = y/² x²) represents the distribution of ys under these conditions.
We consider the following example shown in Figure 8. assumed that yr {Cat, and y (CockerSpaniel,
EgyptianCat). The Downstream samples are
Dog), (12, Dog), (Iz, Dog). (z1, Cat)) e XT. YT.
If the reprogrammed predictions calculated by fore (fin (z;; 0)) are
EnglishSpringer,.a oogerspanii....ggggggiiaaageeeee EgyptianCat), { CockerSpaniel, Ø2 CockerSpaniel,
16

---

then the joint distribution p\ = , = yd x can be estimated as a matrix with the following nonzero values:
y
p( = Dog, Y CockerSpaniel xx 2
p( Dog, ys EnglishSpringer x²)
p( = Cat, Y EgyptianCat x²) 4'
as is shown in Figure 8. Similarly, p(Y'S = = x( can also be estimated.
D Detailed Procedures of Output LM Methods
This section provides a detailed exposition of gradient-free LM methods. Such methods, derived from data distributions, obviate the need for gradients in the output mapping phase. The pscudocode is presented below. Similar to Section 3, w represents the one-to-one LM or probabilistic LM.
D.1 Random Label Mapping (RLM)
Algorithm 3 Random Label Mapping for VR
1: Input: Pretrained label space y with ky labels, downstream label space yT with ky labels
2: Output: One-to-one LM w {0, 11.11ka xkr
3: Initialize <- {0] x temp set T +- (} to store matched pretrained labels
4: Computing output mapping w
5: for y" y do
6: Randomly select S E y -
7: Wyyh.aT
8: TU-U
9: end for
10:
return
The process of random label mapping (RLM) is outlined in Algorithm where the computation of w does not involve the downstream training set. The algorithm establishes a random one-to-one mapping between the pretrained and the downstream labels, ensuring that each y corresponds unique S RLM is computed once before learning the input VR (:;
D.2 Frequent Label Mapping (FLM)
Algorithm Computing Frequency Distribution Matrix of [predicted pretrained label, ground-truth downstream label]
1: Input: Downstream training set {<x]] ) given input VR fin(;)) and pretrained model fpre(-) with the jth dimension being fpre(.).
2: Output: Frequency distribution matrix d E Zaaakr
3: Initialize d kg
4: # Computing frequency distribution matrix d
5: for i I...n do
6: yi + arg maxx (fr ());))))
7: dy. ".. x dz. ",
8: end for
9: return
The procedure of frequent label mapping (FLM) outlined in Algorithm 5. Initially, it utilizes the pretrained model to obtain predicted pretrained labels for samples of the downstream task. Subsequently, it computes joint distribution matrix between the predicted pretrained labels and the ground-truth downstream labels. Finally, employing greedy algorithm, it iteratively identifies the
17

---

Algorithm 5 Frequent Label Mapping for VR
1: Input: Pretrained label space y with his labels, downstream label space y with ky labels,
downstream training set {(2} y/ }}==> given pretrained model (-)e())
2: Output: One-to-one LM w E {0, jkg
3: Initialize w + {0) x temp set T < 0 to store matched pretrained labels, initialize f:n(;)
(0 <- 0)
4: # Computing frequency distribution matrix d
5: Use Algorithm 4 obtain
6: # Computing output mapping w
7: while size of 7 is not ky do
8: Find the maximum dy, y in d
9:
wyys."
10: dyss. - 0 for / = 1, .... k
dz. 0 for 1, 2, .... Kg
T TU {y8}
13: end while
14:
return w
maximum value in the rows and columns corresponding to unmatched label pairs in the matrix to determine the one-to-one mappings. FLM also computed prior to the training of fin (:)
D.3 Iterative Label Mapping (ILM)
Algorithm 6 Iterative Label Mapping for VR
1: Input: Pretrained label space y/8 with ky labels, downstream label space y with ky labels, downstream training set {xx] y/ J given pretrained model fpre(-), total iteration number
E. learning rate a
2: Output: One-to-one LM matrix w E {0, 1] kgkkr
3: Initialize w +- {0] ks x temp set T < 0 to store matched pretrained labels, initialize fin(;)))
(0 0)
4: for 1... do
5: # *Computing frequency distribution matrix d
6: Use Algorithm 4 to obtain d
7: # Computing output mapping w
8: while size of I' is not ky do Find the maximum dy in d
9:
10:
w +- 1 dys. + for
2, ..., br de,y < 0 for = 1, ks
12:
13:
TU T-TU
14:
end while
15: # TTraining fin(: (-;
16: O +-- a. V Σn Elv." Sor (f :))))
17:
end for
18: return W
As enhanced version of FLM, iterative label mapping (ILM) employs interleaved updates with at each epoch, as outlined in Algorithm 6. Such interleaved updates take into consideration the variations in the output space induced by updates the input VR during the training process, thereby ensuring that the output LM will be matched with the updated VR.
D.4 Bayesian-guided Label Mapping (BLM)
The detailed procedure of fayesian-guidle label mapping (BLM) proposed in this paper is shown in Algorithm 7. Compared to ILM, BLM replaces the previous one-to-one mapping {0, 1) flask
18

---

with probabilistic LM w E [0,1]MexAr; both satisfying Σ wj.- = as stated in Section 3. Meanwhile, the process of matrix computation for BLM is based on the Bayes' theorem (detailed in Section 4) reflect the complex relationship among label spaces, rather than determining the optimal match through the oversimplified greedy algorithm.
Algorithm 7 -ayesian-guuded Label Mapping for VR
1: Input: Pretrained label space y with ks labels, downstream label space y with kT labels, downstream training {(=] 3/5" 1n given pretrained model fpre()), total iteration number
E. learning rate a, Laplace smoothing X
2: Output: Probabilistic LM E [o, 1] x
3: Initialize w <- {0)*s x initiialize (: (0 4- 0), temp matrix P [Pt,..g Pkgl Rs
4: for e e= 1...
5: # Computing frequency distribution matrix d
6: Use Algorithm 4 to obtain d
7: # Computing output mapping w kr 8: Pys + Σi=1 dy + for y = 1...ks
9:
w NT + dya.r// /Tt/P. /Pys s for y = 1.., y² = 1.r
10:
# Column normalization of
11:
Wi NT + Wyy w Ex= WoyT for ys 1...ks.. kik 1rt 1...br
# Training Sim (:n(;;).
13:
O +- -0 a Vo >E ((i Sou ;0)))
14: end for
15: return
D.5 Improved Bayesian-guided Label Mapping (BLM+)
Algorithm 8 Computing Probability Aggregation Matrix by Top-K K Predicted Probabilities
Input:
Downstream training set {(x,] y))) )= given input VR (- 0) and pretrained model
f-re(-) with the jth dimension being (.),. Laplace smoothing X, top-K value k:
2: Output: Probability aggregation matrix d' E kR X
3: Initialize d' {0) 10)ks temp matrix Q [Q 1, Qal E RR. K = [K1, -,.... Ki]T NN+
4: # Computing aggregation distribution matrix
5: for 1...n do
6: Q < TopK, (softmaxaa (fin (x: k)) top-K
7: K TopKIndices (f fi (IT: 0)), k)
8: d'x d'x.9.. d' d'K....TT + Q for = 1...k Probability Aggregation
9: end for
10: return
As mentioned in Section 4, BLM+ extends BLM by incorporating the aggregation of top- predicted probabilities, shown in Algorithm 9.
This divergence manifests in the computation process of the joint distribution matrix between predicted pretrained labels and ground-truth downstream labels. Previous methods (i.c., RLM. ILM, BLM) computed a non-nege non-negative integer matrix E x based on the frequency of occurrence of samples (Algorithm 4).
In BLM+ the calculation entails replacing the deterministic frequencies with predicted probabilities from the top K pretrained labels to estimate the joint distribution matrix d E R as is shown Algorithm 8. In the procedure, the probability aggregation substitutes the binary frequency distribution (0. 1} with probability distribution within the range of (0. 1]. while the top-K technique serves retain the most probable predicted labels rather than selecting only one (i.e., BLM) or all labels (denoted as -Top-K' in ablation studies in Section 5).
19

---

Algorithm 9 Improved Bayesian- guideed Label Mapping for VR
1: Input: Pretrained label space y with kis labels, downstream label space y with ky labels, downstream training set { 3/
} given pretrained model fpre(-) with the jth dimension being (pre()),. total iteration number E, learning rate a, Laplace smoothing top-K value k
2: Output: Probabilistic LM E [0, 1 1ks
3: Initialize w <- {0}*e xkr, inittialize f:al;; 0) (0 + 0), temp matrix P [P1, Payl I R
4: for e 1...F. do
5: # *Comutiing probability aggregation matrix d'
6: Use Algorithm& 8 to obtain d'
7: # *Computing output mapping w
8: Pus + ΣiE dyp,
for y 1...ks
9:
Pos # Column normalization of
WyS.T + dya / Pag for of = kis 112 = L...T
10:
11:
wyy TT + wy,s /TT/ Ex=1 WayT for y L...kss 11 k...rr
12: # Training finl: (=)(;))
13:
0 <- 9 a V 2n E(V, Sou (Frro
14:
end for
15:
return w
D.6 Quick Version of ILM, BLM, and BLM+
The haseline method FLM calculates the mapping w once and keeps it fixed, while ILM and our methods update w at each step. However, updating w does not require running the model twice obtain current predictions for each epoch. Instead, predictions from the most recent epoch can be reused. Therefore, only the first epoch is it necessary to run the pretrained model an additional time to initialize the weights of LM, which is the same as FLM. In subsequent epochs, these methods do not require any extra runs. More details can be found in the quick version of our released code.
E Detailed Theoretical Analysis
E.1
Justification and Analysis
In this section, we investigate why probabilistic LM should be favored over deterministic one-to- one mapping. This analysis assumes the existence of true correspondences between labels in the pretrained and downstream domains. We establish that, under certain conditions, probabilistic LM (Definition. E. outperforms deterministic LM (Definition E.2) in estimating the distribution of true label correspondences, quantified by the expected accuracy of the LM function (Eq. (10)).
This analysis focuses on the comparisons of LM. Given that the pretrained model fpre. input x, and input transformations fin are the same across different LM methods, we will omit these notations below unless explicitly needed. We begin by introducing key definitions.
Definition E.1 (probabilistic label mapping (PLM)). Let Fptm Fim be set of mapping functions such that for all fplm E FFimm we have
= = - = uy s.f. Σ - Y.T
(11)
eeys
Here, p(fptm (yS) = y TT (18) is the conditional probability that a pretrained label yf mapped downstream label y²
Definition E.2 (deterministic label mapping (DLM)). Let Falm C Fi be a set of mapping func- g(y²) for all y E ys, where g(y²) specifies a deterministic rule, either )(yو)) ) = S for identity mapping: or g(x²) =1 for flip mapping, respectively. Then, deterministic label mapping defined as: Vfalm E Falm.
tions, defined by falm(y55) falm
{1
if of =
plfarm (v - 1 - "( with "g
(12)
otherwise
where 8 is the Kronecker delta function, ensuring Tt is uniquely mapped from a pretrained label ya.
20

---

Then, we demonstrate the conditions where fpc(f)lm) 2 Aco(falm) Since DLM is defined by following either identity mapping or flip mapping exclusively. each case will be discussed separately.
Lemma E.3. Given a collection of paired labels {(y,: (uu.y=))' -= If the aggregate conditional probabilities ply 1ly* = 0) 2 ply Oly? = 0) and p(y = Oly" = =1) 2 p(y² = 1ly² 1) hold true, and considering falm is defined by identity mapping as outlined in Definition E.2, then it follows that Acc(fplu) 2 Acc(falm)
Lemma E.3 (proof in Appendix E.2) implies that PLM achieves at least as high expected accuracy as DLM defined by identity mapping, under the following conditions: for downstream samples with y² O, the inequality is satisfied when they are more likely to correspond to pretrained samples with y 1 than those with y² 0; for downstream samples with y = 1, the inequality is satisfied when the corresponding pretrained samples are more likely to have y = 0 than o 1.
Uncertainty in Label Inter- deter-Dependenciss. Essentially, the conditions above reflect potential complex patterns of label correspondence that arise when inter-dependencies between the labels exist across domains. While this "label mismatch" problem has been discussed in binary settings, it can be generalized to multi-class settings without loss of generality. Unlike DL.,, which merely relies on static mapping rule and hence may fail when true label correspondence conflicts with this predefined rule, PLM captures the conditional probabilities of y given y$. By harnessing the inherent uncertainty encoded in the probabilistic form of w, PLM expected to achieve more robust label mapping predictions.
Next, we compare PLM with DLM using the flip mapping rule.
Lemma E.4. Given a collection of paired labels {(y y ) 1= If the aggregate conditional
probabilities plys = 0ly = 0) z ply² = 11y = 0) and ply = Oly =) S ply = =1yy² 1), and faim is defined by flip mapping as outlined in Definition F.2, then Acc(Spim) 2 Aco(falm)
Lemma E.4 (proof in Appendix E.2) establishes another sufficient condition under which PLM could achieve an expected accuracy least as high as DLM defined by flip mapping. The condition applies to all downstream samples, regardless of their labels (both y² = 0 or y = 1), stating that it is more likely that their corresponding pretrained label being y rather than y = O.
Bias in Label Correspondences. The bias in Label correspondence refers to a phenomenon where a disproportionate number of downstream samples correspond to pretrained samples with a specific label. For example, consider a medical diagnosis task where both pretrained and downstream data come from populations with low disease prevalence, the label correspondences may exhibit this bias [55]. While this bias may be overlooked by DLM, it could be captured and even exploited by PLM. which flexibly adjusts the weighting schemes, e.g., assigning higher value to 11,0 than wo,o for samples where y O, and to W1,1 over wo,1 for samples where y² =.
Corollary E.5. Let fplm and falm denote the label mapping functions defined in Definition E.1 and Definition E.2, respectively. Given pretrained and downstream label spaces ys {0, 1) and y 1), for any joint distribution over yd x y²T,
3a E 10 s.t. p(y = aly" = a) 2 plu = =dy= =a)
(13)
where à is the opposite label of a, then we have Apc(Spim) Z Ace(Jamm)
Remark E.6. Corollary E.5 implies theoretical foundation for preferring PLM over DLM in scenarios where the label mapping relationship between two domains is uncertain, biased and potentially deviates from a deterministic one-to-one mapping assumption. This finding holds importance in label mappings for VR, as the label spaces may encompass multi-class settings. Furthermore, in VR settings, the pretrained labels derived from fpre predictions, are subject to increased uncertainties and biases influenced by the quality and distribution of the pretrained model and dataset
E.2 Completed Proof of Lemma E.3 and Lemma E.4
Lemma E.7 (ef. Lemma E.3). Given a collection of paired labels {(yS y )) the aggregate
conditional probabilities p(y² = 1ly = 0) Z p = 0ly = 0) and p = 0,y" = 1) 2 p(y²
'For the analysis purpose. in this section we simplify the setting and operate with ground-truth y In practice, VR does not have access true y but must rely on the predicted ye from the rell-trained fpee instead.
21

---

11y" = 1) hold true, and falm is defined by identity mapping outlined in Definition E.2, then Acc(f 2 Ace(faim).
Proof. Expand Eq. (10) by taking possibilities of y², we have:
Amc(fim) EyTEyT
Σ EySES
ply") p (Jimly)) =111²²)) (w9))
Σ p(v²)
E
p(u (fmm 11 ,T4)
(14)
FTEYT
VPEYS
Σ FEEYT
plv")
Σ yeess
p(u (Jim (1)5))
Note that the conditional independence holds since the output of fim relies solely on the input y² For DLM defined by identity mapping, p(famm(y (3²) y" (ly))= = 1 if yd y", and otherwise. Taking into account all the samples, the expected accuracy Ace(famm can then be expressed by
Acc(falm)
Σ p(y") (ly))((s² =y-1ly²)
EYETT
(15)
-p(y² 0)p(y = =0y 0)
plu =)
1)p("" = 111 1).
As for PLM, the expected accuracy can be rewritten as
Aco(Spim)
Σ p(x")
Σ Wy p(y"(1²))
8" ""E
"EyS
(16)
= =ply* = 0) [Wo, p(y² = 0ly² =0) w1,0 ply² 11yy = 0)]
+ ppy = 1) [Wo.1 p(y = 0ly² 1) + w1,1 ply² 11y² = 1)]
where Wo,0 stands for wys==0===0 and similarly for the remaining w W1,0: W1,1-
To evaluate the expected accuracy of fplm and falm. we look into the comparison separately for each yT Specifically, for the samples with y/ = 0, we aim to show that
ply T 0) [wo,o p(y² 0ly" 0) w1.0
ply² = =11y² =1yy"'=0 =0]
(17)
2 p(y T 0)p(y 0ly 0).
Given the constraints wo,0 + w1,0 = and ply 0gy = 0) + p(ys = -1yy = 0) = 1, the LHS of Eq. (17) becomes
p(y 0) [Wo, ply 0ly² 0) + w1,0 ply 11y" 0j]
p(y T 0) [Ww0, p(y" = 0 = 0) - w1 p(y = 0y = 0))
(18)
PLy 0) [Wo,o - w1,0) ply 0ly² = 0) + w1,0
The inequality we need to show is then simplitied p'y" [) - w1,0) p(y 0ly
0) wi,0] 2 p(y 0)p 0ly 0). This inequality holds if p(y 0ly 0) ply²
1 = 0). Similarly, for samples with TT 1, the inequality of interest
ply = 1) [w1,0 p(y = 1ly = 0) + W1,1 p(y = -1y = =1)]
(19)
2 p(y" = 1)p(yS 1 =).
This holds if p = 11y 1) ply = 0ly 1).
Both conditions can be satisfied without conflict. Thus, we can confirm Lemma E.3 by evaluating these conditions jointly.
22

---

Lemma E,8 (cf. Lemma E.4). Given a collection of paired labels {lu" yy )}" the aggregate conditional probabilitiess p(y = 0ly² = 0) S p(y = 11y² = 0) and ply² = -ly² = 1) S p(y =
1y"" = 1), and falm is defined by flip mapping as outined in Definition E.2, then Acc(fplm) Z Acc(falm)
Proof. When defined deterministically by flip mapping, DLM can be equivalently expressed as paam (y²) = y² ( if + y² and 0 otherwise. This allows the expected accuracy of DLM to be expanded
Acc(falm) = Σ
puy p(y")((() ply # fy"1x²)
V"EyT
r(y)) ply") - (1- = ()))
(20)
V"EYT
-ry"" 0) p(u = -1y² = 0)) + + 1) p(5 00
Meanwhile, the expected accuracy of PLM remains consistent as in Eq. (16). Again, to show that Acc(Jpim) 2 Aco(Salm) holds, we compare the expected accuracy with respect to different y²
samples separately.
For samples with y² = 0, we need to show
ply T = 0) [wo,0 p(y = 0ly² = 0) + w1,0 ply² = 11y = 0)]
(21)
2 p(y" = 0)p(y" = 0) (1 p(y = =1ly =0)]
Given the constraints that 40,0 + W1.0 and p(u 1yy 0) = 1 plu 0ly = 0), the
LHS of Eq. (21) can be expressed by
p(u 0) [wo,0 ply 0ly T 0) + w1.0 ply =1yy =0]
(22)
= ppy=== = 0) [( w1,0) ply 0ly 0) + w1,1|
We rearrange the terms:
p(y =0) [(00, - w1.0) ply = 0yy T = 0) + wn.0] 2 p(y = 0) (1 ply = 0ly = 0))
(Wo0 w1,0) p(y 0ly = 0) + w1. 2 ply² = =0ly 0)
1 ply = 0ly TM 0) w1.0 W1.0 ply 0ly" 0) < W0.0
plys 0ly = 0)
1 ply = 0ly² 0)- W1,0 (1 p 0ly = 0]) < wo.0
p(yS = 0ly = 0)
(1 w1,0) (1 ply 0ly = 0])
S Wo.0-
p(yS = OςyT = 0)
(23)
is then concluded that Eq. (23) holds if p(y = 0ly = 0) S ply = 1ly = 0).
As with TT = 1 samples, similar derivation is performed to satisfy the inequality
p²y" T = 1) [wo,1 -pys = 0ly² = 1) + w1, p(y = 11y² = 1)]
(24)
2 plu T 0)p(y" = 0) (1 p(y = 11y 0))
Resembling T 0 samples, the derivation yields the condition p(ys
011 = 1) S plus
1ly" = 1).
Notably, the condition plyS = 0ly = 0) s p(y = 1ly = 0) does not conflict with p(ys
0yT 1) S p(y 1ly" = 1), and both conditions can jointly satisfied.
F
Training Details
F.1
Dataset Information
Additional dataset information is presented in Table 4. For fair comparison, we adhere to the data partitioning scheme employed by IIM [4] through all datasets. The batch size for Oxfordpets and DTD is set to be 64 while 256 for the remaining datasets.
23

---

Table Detailed dataset information
Dataset
Original Image Size
Training Set Size
Testing Set Size
Number of Classes
Hlwers1122
128 x
128
4,093
2,463
102
DTD
128 128
2,820
1,692
47
UCF101
128 x 128
7,639
3,783
101
Food101
128 128
50,500
30,300
101
GTSRB
32 x 32
39,209
12,630
43
EuroSAT
128 X 128
13,500
8,100
10
OxfordPets
128 128
2,944
3,669
37
StanfordCars
128 X 128
6,509
8,041
196
SUN397
128 128
15,888
19,850
397
CIFAR10
32 x 32
50,000
10,000
10
CIFARI00
32 x 32
50,000
10,000
100
SVHN
32 x 32
73,257
26,032
10
F.2 Parameter Information
Consistent training settings are maintained to ensure fair comparison. For training input VR patterns, we apply the Adam optimizer [26] with an initial learning rate 0.01. The number of epochs is 200, with the learning rate decay being 0.1, scheduled epochs 100 and 145. All experiments are conducted on single A100 GPU and the average accuracy three distinct random seeds are reported.
G Parameter Analysis
G.1 Choosing Hyper-parameters
Table 5: Tuning ratio and Laplace À (ResNet-18, 18, Flowers I (2, average accuracy (%)
a|A
0.01
0.1
10
1000
0.01
40.5+0.8
41.7+1.4
44.1+0.1
45.1+0.6
42.9+0.4
40.5+0.4
0.05
46.2+0.4
45.8+0.8
48.9+0.2
47.2+0.4
45.2+0.8
43.0+0.1
0.15
48.2+0.4
49.4+1.0
50.1
48.140.6
45.4+0.7
44.6+0.2
0.1
48.640.8
50.0+1.0
48.4+0.4
48.4+0.6
45.8+0.9
45.6+0.5
1
49.1
50.2+0.2
49.3+0.7
49.3+0.6
45.3+0.7
44.4+0.7
Average
46.520.6
47.440.9
48.1
47.640.5
44.940.7
43.640.4
As described in Section 4, the ratio is used in calculating |a kr). The experimental results to tune
a and À are reported in Table 5. is chosen among [0.01, 0.05, 0.15, 0.5, 1). while is chosen among [0.01, 0.1, 1, 10, 100, 1000). The optimized À is determined first to by the average accuracy of different values, followed by deriving an optimal a 0.15.
While the same hyper-parameters may not necessarily be optimal across different datasets, for the sake of consistency and fairness, this paper employs identical hyper- parameters for all datasets.
G.2 Analyzing Hyper-parameters
Figures 9 and Figure 10 illustrate the impact of À and accuracy. It is observed that the optimal hyper-parameters vary across different datasets.
In general, increases, the test accuracy initially rises and then declines. This parameter is used to balance the contributions of individual pretrained labels. An over-small À might overly rely on the distribution of pretrained labels obtained from pretrained models, while a too- one might overlook differences among pretrained labels. Meanwhile, with an increase in a, accuracy first increases, then platcaus slightly decreases. This is because excessively small large values may

---

Top
Luplace
Top Ratio 0.05
Top Ratio 0,50
Laplace 0.1
Laplace 10
Laplace 1000
Performance Using Different Laplaco- - 10o
Performance Using Different Top Ratio Flowerss
50.0
47.5
47.5
Ascuneyy
Accurecy
45.0
45.0
42.5
42.5
40.0
40.0
10
10
10
10
10'
0.4
0.6
0.8
Laplace
Top Ratio
Performance Using Different Top Ratio OxfordPets
Using Different Laplace- OxfordPets
72
72.0
Accuracy (56)
Accuracy (5)
70.0
68
68.0
66-
66.0
10"
10
Laplace
Ratio
Figure Accuracy with different Laplace A.
Figure 10: Accuracy with different Ratio a.
lead to the neglect of certain crucial labels or the emphasis on redundant ones during the estimation of the probability aggregation matrix. Therefore, choosing moderate values for À and a appears be more appropriate.
G.3 Task-specific Hyper-parameters
Table 6: Difference between task-specific parameters and shared parameters
Flowers102
UCF101
DTD
OxfordPets
CIFAR10
Specific
0.15
0.15
0.5
0.5
0.5
Specific X
10
10
Accuracy (Trained on 70% Samples)
45.82
31.84
43.75
72.27
66.54
Shared a Shared A Accuracy (Trained on 70% Samples)
0.15
0.15
0.15
0.15
0.15
1
45.82
31.84
42.31
70.52
66.04
We used universal hyper-parametery show that BLM and BLM+'s performance gains baselines are not sensitive to hyper-parameterr However, we assume that the dataset-specific tuning for hyper-parameters could yield more optimized results.
Additional experiments are conducted using a validation set and training set split of 30% and 70% to find optimal hyper-parameter for each dataset. Results are shown in Table 6. We observe that optimal hyper-pann parametrs tailored for each dataset achieve better performance compared to using shared hyper-paramette which matches our assumption.
H Limitations of BLM and BLM+
Less effective for tasks with very few classes. As shown in Table 1, when the number of classes (i.e., size label space) in downstream tasks is smaller (10 classes SVHN and classes in EuroSAT) and the original task is relatively simple, the advantage of BLM and BLM+ is not very pronounced. This because BLM and BLM+ replace the one-to-one mapping with a pairwise- connected probabilistic LM. While this optimization yields positive results in most tasks, for
25

---

small subset of simple tasks, the one-to-one mapping may better reflect the relationship between the pretrained label space and the downstream label space. For such tasks, BLM and BLM+ longer exhibit significant effects.
Not solving the cases where VR is not applicable for downstream tasks. For example, in the of the StanfordCars dataset in vision models, as shown in Table and Table 2, the accuracy of the downstream task remains consistently low (<10%) through learning using input VR. While applying BLM and BLM+ in such scenarios yields better results compared to using one-to-one mapping, it still cannot significantly enhance VR performance to the extent of being comparable to finetuning the entire model.
I Visualization of Label Mapping Matrices
Mapping Matrix Visualization (Gradient-free BLM)
Mapping Matrix Visualization
(Grudient-Irre BLM
Pretrained Labels ImageNet
Pretrained Lahels- ImageNet
0
0.03
0.03
Prdzained Labdd- .. IntaeNa (with the finsd 10 closss shown)
2
4
0..02
0.02
6
Dovrraatm Label
0.01
0..01
8
g.
0.00
0.00
Downstream Labels CIFARIO
Downstream Labels CIFARIO
Mapping Matrix Visualization (Deep Learning)
Mapping Matrix Visualization (Deep Learning Sigmoid)
CIFAR1O
Pretrained Labels ImageeNtt
Pretrained Lahels ImageNer
0
0.8
2.
0.6
P
6
0.4
8
0.2
Downstream Labels CIFAR10
Downstream Labels CIFARIO
Figure 11: The visualization results of the LM matrices. Using the example of ResNet-18 pretrained on ImageNet-1K IK applied to the downstream task CIFARIO, the left figure displays the first 10 rows and 10 columns the LM matrices (including the result matrix of the first 10 pretrained and downstream labels), while the right figure presents specific labels. Compared to gradient-free LM methods (i.e., BLM and BLM+), deep learning-bast based methods (i.e., single-layer unrestricted neural network & and a single-layer neural network with Sigmoid [0, 1] xgxk demonstrate less interpretability in revealing the relationship between labels.
Based on the example of ResNet-18 18 pretrained on ImageNet- 1K applying to the downstream task CIFARIO, Figure 11 depicts the visualization results of LM matrices. The first in Figure shows the results of gradient-free methods BLM and BLM+, while the second row shows deep learning-based methods which learn linear neural network for föu 'Deep Learning' refers single-layer neural network without constraints (i.c., I2 (akkyxkr) while 'Deep Learning Sigmoid' refers to applying the Sigmoid function to restrict w E [0, 1g x aligning with the range of WBLM and WBLM+. The right part of Figure 11 depicts the specific pretrained and downstream labels corresponding to these matrices.
It is observed that BLM and BLM+ are good at revealing similarities between pretrained and downstream labels. For example, for the downstream label 'Airplane', which visually resembles 'Great White Shark', 'Hammerhead' and 'Stingray', the weights in WBLM WBLM+ tend to be
26

---

higher. Conversely, for dissimilar labels like "Truck' and 'Ostrich', the weights will be approaching However, the weight matrices obtained from deep learning based methods fail capture such clear-label relationship. The results demonstrate the advantages of BLM and BLM+ in terms of interpretability.
Training Cost Analysis
Table 7: Impact of epoch numbers on different label mapping methods
BLM
BLM+
ILM
FLM
Epochs
60
100
150
200
60
100
150
200
200
200
Average Accuracy
44.5
45.2
45.5
45.3
45.8
46.4
46.9
46.7
40.6
37.2
on 12 Tasks (%)
Table 8: Training cost analysis of LM & VR and none-VR finetuning (on Flowers
Deep Learning- based LM
Gradient-fres LM
Finetuning
FLM
ILM
BLM
BLM+
BLM"
BLM+
Linear
Fully
Back- propagation when Learning LM
No
No
No.
No
Yes
ResNet-18
Parameter Number (M) Whole Time (min)
0.10
0.10
0.10
0.10
0.10
0.10
0.20
0.51
11.7
11.97
12.04
11.95
13.06
6.03
6.52
12.34
14.03
15.28
ResNeXt- 101
Parameter Number (M)
0.10
0.10
0.10
0.10
0.10
0.10
2.0
88.8
Whole
24.68
24.81
24.51
24.71
12.33
12.44
24.80
24.49
35.07
Time (min)
The Required Number of Epochs. Different label mapping methods require varying numbers of epochs converge. We initially used 200 epochs with [4] ensure fair comparison with the baseline methods. Additional experiments are conducted to assess the impact of different epoch numbers from [60, 100, 150] on our BLM and BLM+ model, using ResNet-13 18 as the pretrained model. The results are shown in Table 7.
We found that running 100 epochs yields results comparable to those achieved with 200 epochs. This demonstrates that BLM and BLM+ require less convergence time, highlighting their efficiency.
Overall Time Consumption. Table 8 presents comparison of different output mapping methods in terms of computational resources, utilizing the Flowers 102 dataset as the downstream task. Gradient- free LM refers to estimating output mappings using statistical methods, while deep tearning-basee LM treats label mapping as a single linear layer neural network attached after the pretrained models. 'BLM*' and BLM refer to training with only 100 epochs as is shown in Table 7. It should be noted that the running times for ILM, BLM, and BLM+ are measured using the quick version (see Appendix D.6 for details). Apart from VR methods, which fix the pretrained model, the time costs associated with directly finetuning pretrained models are also listed. Here, the term 'Linear' refers to finetuning the final layer of the pretrained model, while 'Fully' refers finetuning the entire model.
Besides, regarding the performance finetuning methods on downstream tasks compared with VR, please refer to [4] for more discussion. Since we mainly focus on LM methods for VR in this paper, which has different problem setting with finetuning, the performance comparison VR and finetuning will not be addressed here.
27

---

We therefore analyze the efficiency of BLM and BLM+ from three perspectives:
Extra Consumption of Calculating the Mapping Matrix Compared with One-to- to-One--oooooooo Map- ping: Compared to the baseline method ILM, the additional cost for BLM and BLM+ primarily involves the gradient-free multiplication and division within the mapping matrix (which is sized according to the source and farget label spaces, 1000 102 in this case). This additional cost is minimal, as shown in Table
Time Consumption of Updating the Mapping Matrix per Epoch: Compared with FLM,, updating w in ILM, BLM, and BLM+ does not require running the model to obtain current predictions for each epoch. Instead, predictions from the most recent epoch can be reused (see Appendix D.6). As a result, there is no noticeable time overhead for updating w per epoch, indicated by Table
Time Consumption of LM and VR Compared with Deep Learning- Methods: It is observed that methods based on deep learning introduce a substantial number extra parameters (which would further increase with larger downstream label space and higher pretrained model complexity) along with the necessity backpropagation for gradient computation. Conversely, the gradient-free LM methods along with VR emphasized in this study do not encounter these challenges.
K
More Results on Visual Classification Tasks
Applying
Downstroam
Top Weiatted Applying BLM-
Top Weighted Applyin
Sea
Flwas ICC
Globe tlisde
urchiz
Parcupine
Jnckfner
Floves102
Mirqpli
Carhfoosr
Jnceinet
Thocapple
Caalifower
Jackiher
Trenic holl
0,2386
0.10K2
Or
Flowars 112
Guania
squacs
Pescil dempener
Arom spaad
Figure 12: Label mapping results of ILM, BLM, BLM+ for VR on Flowers 102 dataset.
Figures 12-16 illustrate the visualization results of label mapping using ILM (one-to-one mapping), BLM, and BLM+ for VR on various datasets with pretrained ResNet-18. For BLM and BLM+, the top three contributing pretrained labels corresponding to the downstream label are presented, along with their respective weights.
Results when the pretrained and downstream labels exhibit appearance resemblance. Figures 12 and respectively depict the outcomes on Flowers 102 and Food101 datasets, each about classifica- tion tasks of various flowers and food. BLM+ is adept at assigning higher weights pretrained labels with greater resemblance to the downstream label in terms of color, shape, and intricate features. In terms of color, evidenced in Figure 15, the top- weighted labels for 'Edamame' comprise 'Green Snake', 'Artichoke' and 'Green Mamba", all sharing green hue. Regarding shape, in Figure 12, the "Gazania" with petal stripes corresponds to top weighted labels such 'Banana' and 'Zucchini', which exhibit similar striping patterns. As for intricate features, in Figure 12, the 'Globe Thistle' with mike appearance aligns with top weighted labels including 'Sea Urchin', 'Poreupine', and 'Cardoon', which possess akin prickly characteristics.
Results when the pretrained and downstream labels exhibit similarities in texture. Figure 13 presents the results on the DTD dataset, which pertains to the classification of various textures. Both BLM+ and BLM assign higher weights to labels sharing akin textures. For example, 'Spiralled'
28

---

Applstag ELME
Devasdrrmm
Tay Weighted Pretroined Labes Applying BLM-
Top Weighted Pretralned Labels Applying BLM
0,8824
0,779
6.6647
0,0911
0,0826 x
Ccll
DTD Spicalled
Seal
Corppode
Ood
Corrpede
Seal
8.1824
B-8T3
ILEMUS
8.190N
ILEKTV
8JA879
Hirr
Eibeas
Iby
Kemonddrr
Porogis
Hay
Poroqiee
Bloor
Calliiwerr
Iiiernlope
Ringseck
Dragently
Figure 13: Label mapping results of ILM, BLM, BLM+ for VR on DTD dataset.
Applying ILM
Dovessrrars
Top vighted Protrained Applying II.MI
Top righted Applying
0.IAER
BJ862
Bullyhyee
UCF Beachall piuh
Ballyager
Soanbhant
Mire
Bullplayer
Flegzote
BUTTI
ILM679
0JNTT
0.1233 X
ELF7%6 X
Eleesi aiila:
UCF1 Plaiing pularr
Acoatic
Bugo
Stage
Accentic
Mause
Busio
8,650
0.866 x
0.1249
0.1171
0.8678
Lipeock
Apple
Lipuics
Iilly
22123
Lipstck
iPod
Icc Iolly
Figure 14: Label mapping results of ILM, BLM, BLM+ for VR UCF101 dataset.
corresponds to top- top-weighted labels embodying spiral-shaped entities such as 'Snail', 'Centipede', and 'Coil', while 'Fibrous" aligns with entities possessing a rough and fibrous texture, including 'Hay', 'Komondor', and 'Porcupine".
Results when the pretrained and downstream labels exhibit similarities in backgrounds. Fig- ure 14 illustrates the results on the UCF101 dataset, dataset for action classification. In this task, both BLM and BLM+ tend assign higher weights to pretrained labels with backgrounds environ- ments akin to the downstream labels. For example, action 'Apply Lipstick' often involves presence of a human face; hence, pretrained labels such as applying 'Lipstick', eating 'Ice Lolly', and spraying 'Hair Spray' contribute significantly. Likewise, labels closely associated with BBaseball pitch' include 'Ballplayer' and 'Scoreboard' featuring backgrounds of vast grass fields.
Results when pretrained and downstream labels exhibit inclusion relationship. Figure 16 illustrates the results on the CIFAR10 dataset, which comprises images broadly categorized into main classes, with each category corresponding to several subcategories within the pretrained domain. It is noted that unlike the singular class selection of ILM, both BLM and BLM+ allocate similar weights to multiple subcategories. For example, 'Dog" corresponds to different breeds such as "Cocker Spaniel', 'English Springer', and 'English Setter', while 'Bird' encompasses subcategories including 'Peacock', 'Albatross', and 'Littte Blue Heron'. Hence, the learning framework of BLM and BLM+ demonstrates effective handling of the inclusion relationship between label spaces.
29

---

Devastream
Top Weighted Fretralzed Labels Applying BLM-
Tay Weighted Pretralned Label Applylng BLM
ILMATT
BJRKX
0,1010
0J8862
IJISSS x
Caarmber
Foodlol Edarane
Groen reaks
Arrchole
Crean manth
make
Or
Food100 Crosse teabe
iquan
Express
Buttena
0
BA112
6,911
Wok
Foodlol Bibimbbp
Web
Dusch even
Conkc book
Figure 15: Label mapping results of ILM, BLM, BLM+ for VR on Food101 dataset.
Applylng
Doserdrnarr
Wrighted Pestrained Labels Applying REM-
Weighted Pretrained Labels II.M
querner
Poy-phone
Teep
Toucker
Snowpiew
Shorppnng
Tracrer
Japp
Cr
joket
Figure 16: Label mapping results of ILM, BLM, BLM+ for VR on CIFARI0 dataset.
Applications on Vision-Language Models
L.1 Learning Framework
The distinction between Vision-Language Models (VLM) and vision models lies in (1) vision models take single image input, whereas VLMs take a pair of text and images input; and (2) vision models have fixed pretrained labels, with model outputs being logits, while VLMs lack pretrained labels, with model outputs being the cosine similarity [57] between images and text embeddings. As a result, when applying BLM and BLM+ to VI.M, is necessary to design an input text set to replace the original pretrained labels vision models.
In this paper, we follow a previous work [4] and construct the input texts set by taking the Cartesian product [56] of the downstream label set and the prompt set. BLM and BLM+ can be applied to compute the joint frequency distribution (for BLM) or aggregated predicted probability (for BLM+) between the input texts and the downstream ground-truth truth labels. This enables the mapping from candidate input texts to probable classification results.
The full process of learning WBLM, for vision models or VLMs is illustrated in Figure 17. Besides, the pipeline and algorithm details are the same as BLM and BLM+ for vision models shown in Figure 2, Algorithm 7 and 9.
30

---

Vision Models
7-axis: Frequency or Probahility
Inference Statistics
Fixed Pretrained Vision
Predicted Pretrained
Distribution
Downstream Task
Input Visual
Reprogramming
Label Prob
Model
Probabilistic LM
WBLM WBLM
fm((.)
X-axis: Downstream Label Set y/7
(10) M D
Corresponding
Y-axis: Pretrained Label Set y
Downstream Label
Downstream Task
Input Visual Reprogramming
Vision Language Models
Fixed Pretrained Vision-
Z-axis: Frequency or Probability
Inference Statistics
f.)
Language Model
Cosine Similarity Prob
Distribution
Probabilistic LM
Text Inpul T y (f.
WBLM WBBLM.
Template [t]]m
Downstream Label Set y
(t1,5))
Corresponding Downstream Label
(bm.
X-axis: Downstream Label y
( . )
Y-axis: Text Inpul Set T y
Figure 17: The framework of learning WBLM or WBLM+ for pretrained vision models (upper) or VLMs (lower). As described in Section 4, for vision models, WBLM WBLM+ is derived from the frequency distribution (in BLM) or probability aggregation matrix (in BLM+) where pairs of [predicted pretrained label, ground-tre truth downstream label] are calculated. Nevertheless, for VLMs, the predicted pretrained label replaced by possible text inputs from the Cartesian product of the downstream label set, and the prompt set. The cosine similarities of images and text embedding are calculated VL.Ms to replace the output logits in vision models.
L.2 Performance Results
Table presents the performance BLM and BLM+ applied to VLMs across 12 datasets. For fair comparison, we follow the previous work [1] to employ CLIP the pretrained model and watermarking- based VR with an outer frame size of 30. We utilized an initial learning rate of 40 and Cosine Annealing learning rate schedule [35], with total of 200 epochs. An SGD optimizer with momentum of 0.9 was employed for learning the Input VR. Results without label mapping (denoted by 'None") ) and one-to-one mapping served as the bascline, and the average accuracy computed over three different random seeds.
From the performance results, it can be observed that except for the EuroSAT dataset, which has small number of classes and simpler tasks (this limitation will discussed in detail in Appendix H), BLM or BL.+ achieves improvements across all other tasks. They achieve the average accuracy of 79. 1 and 79.3%, respectively, without increasing the number of model parameters to trained.
This empirical evidence demonstrates that BLM and BLM+ also be effectively applied to VLMs.
31

---

Table 9: Performance comparison on VLMs (mean % std %)
CLIP (ViT-B32)
Baseline
Ours
Method
None
One-to-one Mapping
BLM
BLM+
Flowers102
70.5
75.5+10
76.921.9
76.4
DTD
61.4+0.6
59.5+1.1
60.940.9
61.5+0.3
UCF101
1
67. 67.940.6
72.2+0.2
72.3+0.4
Food101
79.2+0.2
78. +0.3
79.3.0.1
79.4
GTSRB
91,4+0.4
91.3+0.2
91.5=002
90,9+1.0
AT
96.6+0.1
96.5+0.1
96.3+0.1
96.3+0.1
OxfordPets
10.1
86.8+0.6
88.6+0.5
89.0=0.4
StanfordCars
57.9+0.1
55.8+0.1
59.8+0.7
60.3+0.2
SUN397
61.4
60.6.0.1
63.1
63,8+0,2
CIFARIO
94.0=0.2
94.1
94.2=002
1
CIFARI00
75.
74.8x0.1
75,4
75.5+0.3
SVHN
91.3+0.2
91.3a0.2
91.5+0.1
91.7+0.1
Average
77.9
77.7
79.1
79.3
Top Wrighted Inpur Texts Applying Bll M+
Top Weighted Inpot Texts Applying BLM
a jprg corppted photo king protea
0 1129 the embreideeee king protta
0,0954 * corrupted photo of king protea
0.149
comupted photo of
0.0738 a crupped pluto of king protes of
0,1250
rendering the
0.1112 king protea
cropped
king protes
0.1112 close-up photo the king proten
Origia
of king
I-To-l Mapping
ploto of king protes
Weighted Imput Texts Applying BLM+
Weighted Input Tests Applying BLM
photo small gaura
0L 1,0055 photo of cool gaura
0.0576 plotoo the small gaura
0,0804 phoso of the diny gaura
0.0669 a photo of gaura
0,0804 * photo of dirty
jusz comppted
gaura
gaurs
good photo gaura
Origin
this a photo of gaura
1-1a-1 Mapping
photo of small gaura
Weighted Input Applying
Top Welghted Input Tests Applying BLM
photo
0. 1436 photo of the werd fire lily
phuto
0.1276 jpez corrupted photo of the fire lily
0.0522 brgght pluto of the fire lily
0.1276 a dark photo the fire lily
0.0410 bright ploto of fire lily
00335 a bad photo of fire
embroidered fire lily
Origin
this is photo o fire lily
I-To-I Mapping
cliso-up photo the fire lill
Figure 18: Results of ILM, BLM, BLM+ for VR on Flowers 102 dataset.
L.3
Visualization Results
Figures 18-22 show the visualization results of top- top-weighted input texts different datasets applying BLM and BLM+. It is evident that, unlike the single optimal text input in one-to-one mapping, BLM and BLM+ assign different weights many possible deseriptions. For example, in CIFAR10. image of a bird may be described as 'a low-resolution photo of a bird', 'a close-up photo of the bird', or "this is a photo of bird', among others. Such methods alfirm different expressions instead of only one description using one-to-one LM.
These experiments further demonstrate that BLM and BLM+ can be used to enhance the performance of input VR in VLMs while providing reasonable explanations for why input VR in VLMs can effectively work.
32

---

Wrighted Texts Applying Bi M
Weighted Enput Toxts Applying BLM
0.0738 ploto ol large
of diry
photo
porous 0,0732 photo white crusshaccedd
ploto dinty
photo the large porous
0.0732 sculpteee ofa humpy
0,0455 ploclated pioto the porous
0.0732 plato of the dity poreus
Origin
photo of'a parous
Mapping
poreus
Weighted Applying
Weighted
Applying
01024 sculpture of perforated
0.0887 ploto of laree perforated
0,0637 This of perforated
0.U587 sculpture of perforated
candeingg
0 0796 rendering perforated
photo a perforated
photo
0,0466 pioclated photo of perforated
while phoso perforated
Origin
thi photo of perforated
I-Tu-1 Mapping
sculpture perforated
Top Weighted Inpot Tetss Applying M+
Top W Weighted Input T Texts Applying BLM
0.0765 sculptuse of's honeycsashed
O 0.9924 rendering of coneycombed
0.0756 photo of large comeycombed
large honeycombed
0.0741 this photo of honeycomhed
photo honeycombed
photo many honeycombed rendering of hescycombed
0.0525 orgamii honeycombed
0.0628 a origami greoved
Origin
Ois is pluto of honeyevabbed
1-To-1 Mapping
piseelated photo of honeycombed
Figure 19: Results of ILM, BLM, BLM+ for VR based on CLIP on DTD Dataset.
Reweighted Text Embrlddings Applying BLM++
Reweighted Source Classes Applying BLM
0.0504 bowling video game
0,090 bowling
0,0470 the bowling video bowling
game
ploto weidd willing
cirtoon hewllig
0,0548 a photo of cool bowling
0,0376 rendeting a bowling
0.0537 a photo of the cool bowlling
Origia
photo bowling
Mapping
Reweighted Embeddings
Reweighted Classes
0.661 billiards in video game
0.0726 resolution photo of billiards
0,06 10 the billiards video
0.0712 billiards is vidoo
reselution photo billiards
0629 grfffti of billlards
rendition
billiands
0,0500** cartoon billiards
photo diny
Origin
this is photo of billiards
1-Ta-1 Mapping
billiurds video
Top Reweighted Text Embeediags Applying BLM++
Top Reweighted Source Classes Applying BLM
0.0332 phoso of hand to see tennis swing 0.0529 art of the tennis swingg
0.0878 art of tennis swing
0.0060
tennis swing.
0.356 axt of drunis swing
photo hand rennis
reselution photo sming 0.0691 the unbouddzed teaais swing.
photo
swing
Origin
this photo at' sennis swing
I-To-l Mapping
pisclated phuto of the teasis swing
Figure 20: Results of ILM, BLM, BLM+ for VR based on CLIP on UCF101 dataset.
33

---

Wrighted
Texts Applying Bi M
Weighted Texts Applying BLM
0.0225 good photo afa churros dhis
0.0291 of the nice churros
close-up photo
ploto
0,0287 photo of churros
0,0215 of churros
0,0282 good phuto of the churros
0,0215 phoso of cool churres
0.0275 art of churros
Origia
a pluto
Mapping
piuto
Weighted Applying BLMM-
Applying
0,0222 photo of the hamburger
0.0429 the custonn hamburger
0,0219 a photo
hamburger
0.0419 photo of the datty hamburger
hamsbarger
0.0416
the hamburger
photo 0,0214 low resolution photo of the hamburger
hamsburger
hamburger
pexciazed photo hamburger
Origin
Cis is photo of hamburger
1-Tb-1 Mapping
piselated hambarger
Top Weighted Impat Texts Applying Mi
Top Weighted Input Tests Applying BLM
0,0254 bright pluto of the croque madame
0.0340 beight photo the croque modamee
0.0299 this is photo of creque madame
croque
02428 a brigt photo a
madame
my croque
photo
croque
0.0316 a photo of diny croque madame
good photo of croque madame
0.0113 rendering of the croque madame
Origin
this is photo of madame
I-To-I Mapping
bright photo of the creque madame
Figure 21: Results of ILM, BLM, BLM+ for VR based on CLIP on Food101 dataset.
Weighted Texts BLM-
Weighted
0.0365 a low resolution photo a bird
0 0.000
close-up photo of bird
0,036: * close photo of the bird
cool
this
low reselution photo of bird
photo weidd
0 0,0295
bright photo of bird
0,0050 blurry photo o bird
0.0292
cartona bird
Origia
photo
1-To-I Mapping
Jow reselutios bied
Weighted
Texts BLM-
Weighted
0.445 pf small dog
0.0535
photo of small dog
0,0415 cortron dog
0.1335
photo of the cool dog.
resolution
0. 0134
close-up photo of
rceelution photo dlgg cartoon dog
0.022
0,0401
pecelated
dog
Origin
this is photo of dog
1-Ta-1 Mapping
piselated
Top Weighted Iaput Texts Applying BLMM+
Top Weighted Inpot Tests Applying BL.
0.0437
low resolution photo o horse 0.035 piselated photo of the horse
0.0382 blurry photo of horse
0.0382 pisclatel
04229 our photo of horse
resturiinn
0.0376
jaag comppied photo
jaag cocrupted photo
0.0374
pisclated photo of the horse
Origin
iii is photo of horse
I-To-l Mapping
al piselated phuto of the horw
Figure 22: Results of ILM, BLM, BLM+ for VR based CLIP on CIFAR10 dataset.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract are detailed in the introduction part, and further discussed in each section. Each paragraph in the introduction is provided with corresponding link certain sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Appendix H.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaaiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
35

---

Answer: [Yes]
Justification:
All assumptions and proof are included in Appendix E.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All implementation details (described in Appendix F) have been included.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
36

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The link to the code has been provided in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
public/guse swriides/CodeSumiisssse.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss... wttpssccccppcccccccetce.ceccccct http//udddes/oooee /Cooeeuuoiiiseeeeeee ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Detailed dataset information is included in Section 5 and Appendix F. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experiments are three seed with the standard divination being reported in Table 1, 2 and 9, and also shown in strip areas Figure 9-10.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
37

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
Justification: The computer resources have been discussed Appendix and detailed in Appendix J.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ete:///eeurpeee cc wtwplllc/Ethiscuiiddde.o
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Since methods proposed in this paper are used to improve the performance of VR in downstream classification tasks, there is no potential societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
38

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper does not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Creators original owners of code or data have been cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com aamdaatseess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
39

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "Qk32rPHR0F", "reviewer": "Reviewer_RgTa", "summary": "Pretrained models play a crucial role in current machine learning and computer vision tasks, and effectively leveraging them in downstream tasks has become increasingly important. This paper explores the research area of visual reprogramming (VR), which diverges from traditional fine-tuning by adjusting the input space rather than the parameter space of pretrained models. Additionally, VR necessitates establishing a mapping from pretrained labels to downstream labels, which is the primary focus of this paper. The proposed method for mapping pretrained labels to downstream labels is well-motivated and convincing. The experimental results are robust and demonstrate the efficacy of the approach.", "questions": "See Weaknesses.", "limitations": "The authors have adequately discussed the limitations of the work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The experiments are solid and impressive, covering larger models compared to previous works in this field.\n\n2. The motivation is very clear, adhering to basic probability rules.\n\n3. Two methods are proposed based on different estimation approaches for the key component, demonstrating sufficient technical contribution.\n\n4. Some experiments provide insightful explanations on why the mapping learned from the proposed method is effective, which I found particularly interesting.", "weaknesses": "1. Although the motivation is clear, it feels somewhat dry. More detailed explanations are needed for Section 4.1. Specifically, some of the conditional probabilities require further clarification (e.g., Eq. (6)).\n\n2. It is unclear how Section 4.2 can be extended to the multi-class case. The accuracy metric (Acc) used here appears to be suited only for binary cases.\n\n3. While it is commendable that a universal hyperparameter performs well in your experiments, it raises the question of whether these hyperparameters can be tuned using a validation set. How do all methods perform when evaluated on the same validation set?\n\n4. I am not entirely convinced by Figure 4. Why is pineapple selected as well? This needs further explanation.\n\n5. The absence of algorithm tables in the main content is a significant omission and should be addressed. Similar to the first weakness, the current method presentation feels somewhat dry.\n\n6. Is BLM+ without Top-K and Bayes the second column in Table 3? If so, additional descriptions should be included for clarity.\n\n7. What are the fine-tuning results in this context? It is essential to establish a standard to determine when VR is needed. Additionally, what is the total time cost compared to fine-tuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** Thank you. We have added a detailed explanation in **Common Question 2** and provided a simple example to illustrate the conditional probabilities. \n\n**W2:**  Thanks. We want to clarify how our analysis can be extended to multi-class cases.\n\n**Expected Accuracy Definition** The formula (Eq. 10) evaluates the expected probability of correctly mapping each $y^{\\rm S} \\in \\mathcal{Y}^{\\rm S}$ to the corresponding $y^{\\rm T} \\in \\mathcal{Y}^{\\rm T}$, remaining valid even for multi-class settings. It is agnostic to the number of classes in both $\\mathcal{Y}^{\\rm S}$ and $\\mathcal{Y}^{\\rm T}$.\n\n**Mapping Function Revision** Recall the definition of PLM (Definition C.1) and DLM (Definition C.2) are\n- PLM: $\\mathrm{Acc}(f_{\\rm plm}) = \\sum_{y^{\\rm T} \\in \\mathcal{Y}^{\\rm T}} p(y^{\\rm T}) \\cdot \\sum_{y^{\\rm S} \\in \\mathcal{Y}^{\\rm S}} p(y^{\\rm S}) \\cdot \\omega_{y^{\\rm S}, y^{\\rm T}}$\n- DLM: $\\mathrm{Acc}(f_{\\rm dlm}) = \\sum_{y^{\\rm T} \\in \\mathcal{Y}^{\\rm T}} p(y^{\\rm T}) \\cdot \\sum_{y^{\\rm S} \\in \\mathcal{Y}^{\\rm S}} p(y^{\\rm S}) \\cdot \\delta_{y^{\\rm S}, g(y^{\\rm S})}$\n\nFor multi-class cases, we need to revise the mapping function from $f_{\\rm lm}(y^{\\rm S}) \\in \\lbrace y^{\\rm T}, 1 - y^{\\rm T} \\rbrace$ in binary label spaces to align with $\\mathcal{Y}^{\\rm S} = \\lbrace1, 2, ..., k_{\\rm S} \\rbrace$, $\\mathcal{Y}^{\\rm T} = \\lbrace1, 2, ..., k_{\\rm T} \\rbrace$. Thus, we need to expand previously used identity/flip mapping rule $g(y^{\\rm S})$ to cover a broader range of possible mappings.\n\n**Proof Sketch** PLM can achieve at least the accuracy of the optimal DLM by constructing $\\omega_{y^{\\rm S}, y^{\\rm T}}$ to match the optimal deterministic mapping rule $g^*(y^{\\rm S})$, which ensures $\\sum_{y^{\\rm S} \\in \\mathcal{Y}^{\\rm S}} p(y^{\\rm S}) \\cdot \\omega_{y^{\\rm S}, y^{\\rm T}} \\geq \\sum_{y^{\\rm S} \\in \\mathcal{Y}^{\\rm S}} p(y^{\\rm S}) \\cdot \\mathbb{I}[g^*(y^{\\rm S}) = y^{\\rm T}], \\forall y^{\\rm T} \\in \\mathcal{Y}^{\\rm T}$. Intuitively, this inequality holds, especially when $g^*(y^{\\rm S}) \\neq y^{\\rm T}$, $\\mathbb{I}[g^*(y^{\\rm S}) = y^{\\rm T}]=0$, while $\\omega_{y^{\\rm S}, y^{\\rm T}}$ can be greater than 0 in such cases, showing better flexibility.\n\nDue to the character limit, we leave the complete proof to future work. We hope this clarification addresses your concern.\n\n**W3:** Thank you. In Appendix Sec. E, we have acknowledged that optimal values may vary across datasets (Fig. 8-9). Our initial use of universal hyperparameters was intended to show that BLM/BLM+'s performance gains are not sensitive to hyperparameters.\n\nFollowing your advice, we quickly run additional experiments using a 70\\%/30\\% train/validation split of the original training set to find optimal hyperparameters for each dataset, shown as\n\n| | Flowers102 | UCF101 | DTD | OxfordPets | CIFAR10 |\n|------------|------------|--------|-------|------------|---------|\n| Optimal $\\alpha$ | 0.15 | 0.15 | 0.5 | 0.5 | 0.5 |\n| Optimal $\\lambda$ | 1 | 1 | 1 | 10 | 10 |\n| Accuracy (Trained on 70% Samples) | 45.82 | 31.84 | **43.75** | **72.27** | **66.54** |\n| Shared $\\alpha$ | 0.15 | 0.15 | 0.15 | 0.15 | 0.15 |\n| Shared $\\lambda$ | 1 | 1 | 1 | 1 | 1 |\n| Accuracy (Trained on 70% Samples) | 45.82 | 31.84 | 42.31 | 70.52 | 66.04 |\n\nObserve that dataset-specific tuning indeed yields better performance compared to shared hyperparameters, suggesting that optimal hyperparameters tailored to each dataset are desired. We plan to include these in the revision. \n\n**W4:** Thanks for your question. This observation stems from the statistical nature of our BLM+ algorithm, which computes label mappings based on the co-occurrence of predicted pretrained labels and ground truth downstream labels throughout the VR learning iterations (Fig. 4 visualizes the top-3 predicted pretrained label at each iteration).\n\nIn the later stages of VR learning, we observe that for marigold images $X^\\mathrm{T}$, the conditional probabilities shifted:\n$p(Y^{\\rm T}={\\tt Marigold}|Y^{\\rm S}={\\tt Pineapple}, X^{\\rm T})>p(Y^{\\rm T}={\\tt Marigold}|Y^{\\rm S}={\\tt Teddy}, X^{\\rm T})$\n\n$p(Y^{\\rm T}={\\tt Marigold}|Y^{\\rm S}={\\tt Pineapple}, X^{\\rm T})>p(Y^{\\rm T}={\\tt Marigold}|Y^{\\rm S}={\\tt Broccoli}, X^{\\rm T})$\n\nThis indicates that the feature of marigold images, after processing by input VR and the pretrained model, share more similarities with pineapple than with teddy bears or broccoli. Thus, pineapple replaced these other labels among the top-k predicted pretrained labels.\n\nVisually, this replacement is intuitive as well: The colors of a pineapple — primarily yellow, orange, and gold — are similar to those of Marigold. Teddy bear and Airedale are predominantly brown, while guacamole and broccoli are mainly green. Additionally, the shape of a pineapple, which is typically oval, resembles that of Marigold.\n\n**W5:** Thank you for your suggestion. While space limitations led us to initially place the algorithm tables in the appendix (lines 455 and 462), we recognize the importance of making this information more accessible. In the revision, we will prioritize incorporating them into the main text. We will also work on enriching and smoothing out the method presentation to make it more engaging.\n  \n**W6:** Yes, BLM+ without Top-K and Bayes is the same as BLM without Bayes. Thanks for the reminder, we will explain it clearly.\n\n**W7:** Thank you for your questions. As we follow the settings in [1], the fine-tuning results can be directly quoted from [1], and we will include this in the next version of our paper. Regarding training time, please refer to **Common Question 1** for a detailed answer. Regarding the standard to determine when VR is needed, we believe that (1) when issues such as copyright or avoiding catastrophic forgetting exist, the pre-trained model needs to be kept unchanged; (2) when the resources for training downstream tasks are limited, VR can be used.\n\n[1] Chen et al. Understanding and improving visual prompting: A label-mapping perspective. In CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "egsuPoNAZR", "reviewer": "Reviewer_885t", "summary": "Another type of transfer learning approach is considered in this paper: model reprogramming. Different from classical transfer learning approaches, model reprogramming only changes models via changing the input space and output space, which is more efficient to fit a pretrained model to some downstream tasks. Specifically, this paper focuses on the output space, label mapping. The motivation is strong as it is clear that single map does not work well. Then, new methods are proposed based on Bayes' theorem. Experiments are enough and solid, in terms of model size or dataset size.", "questions": "1. Transfer learning literature is missing in this paper. Although model reprogramming is not considered as a transfer learning method in the literature, it is one of transfer learning techniques. This review can help position this paper better.\n\n2. Can we transfer a pretrained model to any possible task? Are there standards to help choose which model should be used for a specific downstream task?\n\n3. For example, if we can use a model trained with cars to help recognize images of animals? It looks impossible. How to tell when we can use a pretrained model?\n\n4. Computing w in BLM also needs time. Is there efficient advantage of VR anymore? What is the performance of finetuning? The gap should be reported in this paper.", "limitations": "No concerns for this paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Transfer learning is even more important in the current era. This paper focuses on an interesting direction, which is significant to the field.\n\n2. The main result of this paper looks general, compared to previous label mapping methods. It is novel to first consider a pretrained model's output into the P(y|x) and then analyze what we should do.\n\n3. I enjoy reading the experiment section, which is quite comprehensive. I did not see some necessary experiment missing.", "weaknesses": "1. Transfer learning literature is missing in this paper. Although model reprogramming is not considered as a transfer learning method in the literature, it is one of transfer learning techniques. This review can help position this paper better.\n\n2. Can we transfer a pretrained model to any possible task? Are there standards to help choose which model should be used for a specific downstream task?\n\n3. For example, if we can use a model trained with cars to help recognize images of animals? It looks impossible. How to tell when we can use a pretrained model?\n\n4. Computing w in BLM also needs time. Is there efficient advantage of VR anymore? What is the performance of finetuning? The gap should be reported in this paper.\n\n5. Main algorithm table should be moved to the main paper. It is not very easy to implement BLM or BLM+ based on the formula, but the algorithms are very helpful.\n\n6. It would be better to explain meaning of some conditional probabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1/Q1:** Thank you very much for your suggestion. In the next version, we will expand our literature review to include relevant transfer learning concepts, particularly focusing on how Visual Reprogramming relates to and differs from traditional transfer learning methods. To better position this paper, we'll also discuss the spectrum of transfer learning techniques, connect and compare Visual Reprogramming with other parameter fine-tuning methods, highlighting its merits in scenarios where pretrained model preservation is crucial.\n\n**W2/Q2/W3/Q3:** Thanks for the question. To make our reply relevant, we will discuss the transferability in the context of VR.\n\n**Transferability in VR** Drawing on theoretical foundations from [1], the transferability of a pretrained model to a downstream task can be bounded by:\n$$\\mathcal{L}^{\\rm{T}} \\leq \\mathcal{L}^{\\rm{S}} + \\mathcal{W}(\\mu^\\rm{S}, \\mu^\\rm{T}) + \\epsilon$$\nwhere $\\mathcal{L}^{\\rm{T}}$ and $\\mathcal{L}^{\\rm{S}}$ denotes error for downstream and pretrained tasks. $\\mathcal{W}(\\mu^\\rm{S}, \\mu^\\rm{T})$ is the Wasserstein distance between the logit distributions of pretrained input $\\mu^\\rm{S} = f_{\\rm pre} (x^{\\rm S})$ and the reprogrammed downstream input $\\mu^\\rm{T} = (f_{\\rm pre} \\circ f_{\\text{in}}) (x^{\\rm T})$, $\\epsilon$ is a small constant. We therefore know that the performance on downstream tasks can be related to the pretrained task performance and the alignment between pretrained task and downstream task.\n\n**Model selection** This bound suggests some insights on selecting models:\n- Pretrained model with higher capacity may lead to lower $\\mathcal{L}^{\\rm{S}}$,\n- Pretrained feature is relevant to the downstream task; or input VR and the output LM that effectively transform downstream input and output (both potentially contribute to lower $\\mathcal{W}(\\mu^\\rm{S}, \\mu^\\rm{T})$ distance, indicating better alignment)\n\nFor example, a model with large capacity pretrained on general object recognition (e.g., ResNet on ImageNet) may transfer better to another recognition task than a model pretrained on a highly specialized dataset.\n\n**Transferring between dissimilar domains (e.g., car-animal)** The feasibility depends on how is the Wasserstein distance minimized - this is theoretically possible even for seemingly unrelated domains, but the practical difficulty varies because of the challenge of measuring domain similarity and feature relevance.\n\n**When to use a pretrained model** Currently, there lacks theoretical tools to accurately measure the above Wasserstein distance BEFORE training, which makes it difficult to directly tell when and whether a pretrained model can be used. Therefore, we look forward to future work that focuses on effective ad-hoc estimation of such distance and techniques that minimize it through optimized VR and LM methods.\n\nIn short, while VR offers a flexible way of transfer learning, the choice of pretrained model and its adaptability to a specific downstream task depends on both model capacity and the ability to align the feature distributions.\n\n**W4/Q4:** Thank you for your questions. For details on training time, please refer to **Common Question 1**. Regarding the fine-tuning results, since we follow the settings outlined in [2], we will directly quote these results from [2] in the next version of our paper.\n \n**W5:** Thank you for your suggestion. Due to space limitations, we have placed the algorithm tables in the appendix (lines 455 and 462). In the next version, we will prioritize incorporating them into the main text to enhance readability and understanding for our readers.\n\n**W6:** Thank you for your suggestion. We have added a detailed explanation in **Common Questions 2** and introduced a simple example to illustrate. We will include this revision in the final version.\n\n\n[1] Yang et al. Voice2Series: Reprogramming Acoustic Models for Time Series Classification. in ICML 2021.\n\n[2] Chen et al. Understanding and improving visual prompting: A label-mapping perspective. In CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jxja04lJYF", "reviewer": "Reviewer_zxuh", "summary": "Visual reprogramming is an interesting way to reuse a pre-trained classifier or an VLM. In previous methods, the way to change the output interface is basically gradient-free and one-on-one mapping. In this paper, the authors found that the previous way is suboptimal and ignores  information. Then, from a theoretical perspective, a new objective is derived using Bayes' theorem. By optimizing this objective, the performance of VR methods is significantly improved, which is demonstrated via extensive experiments. A theoretical study is provided as well.", "questions": "Please refer to \"Weaknesses\".", "limitations": "No further concerns regarding limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research topic is of significance. VR is useful in practice. This paper understands and extends the label mapping from a mathematical perspective, which steps much further compared to previous methods. \n\n2. The derivation regarding w is could generally cover previous methods (which is probably the first). The drawback of previous methods is manifested based on the derivation.\n\n3. Math expressions are easy to follow.\n\n4. Abundant experiments are provided, including large datasets, large models, and VLM. From the experimental results, we can see the improvement is universal, justifying the general effectiveness of the proposed method.", "weaknesses": "1. In Eq. (1), what is the real gap between right and left terms? It seems not easy to get this gap. I suggest using right one directly, which is easy to be acceptable.\n\n2. Above the Eq. (6), one might misunderstand the way to calculate the frequency. Using () might be better.\n\n3. Figure 2 is confusing. It looks like step 3 follows step 1, rather than step 2 following step 1. What is the actual sequence between step 1~3?\n\n4. In section 4.2, the calculation of accuracy is confusing. $x$ is missing.\n\n5. What does $\\bar{r}$ mean in l.262?\n\n6. Training time is potentially a concern. How many training epochs are needed for BLM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** Thanks! The gap between left-hand side (LHS) and right-hand side (RHS) comes from the relationship between Maximum Likelihood Estimation (LHS) and Empirical Risk Minimization (RHS) in statistical learning theory.\n- LHS: represents the true objective of VR is to maximize the conditional probability of downstream label given downstream input image.\n- RHS: provides a practical and empirical approximation using a finite training set and a chosen loss function.\n\nThis formulation aligns with fundamental principles in learning theory: \n- We use empirical risk as a proxy for the expected risk.\n- A loss function is chosen to approximate the negative log-likelihood.\n- This approximation connects to generalization bounds. The gap between the expected and empirical risks can be bounded using techniques like VC-dimension.\n- The law of large numbers ensures convergence of empirical to expected risk as the training set size increases.\n\nHowever, we agree that using RHS directly is more straightforward. We plan to incorporate your suggestion in the next version for easier understanding.\n\n**W2:** Thanks for the suggestion, we will include () in the next version.\n\n**W3:** Thank you for your question. In fact, step 3 (applying $\\omega$) is performed after step 2 (calculating $\\omega$, and the blue dotted line indicates the flow of $\\omega$). However, after calculating $\\omega$, the logits output from step 1 will also be used to obtain the final prediction result in step 3 (as shown by the purple solid line). We will revise Fig. 2 to more clearly illustrate this process in the next version of the paper.\n\n**W4:** Thanks for pointing it out. The omission of $x$ in Eq. 10 was intentional because:\n- Our analysis in Sec 4.2 (and Appendix Sec. C) focuses on output label mapping (LM) and specifically compares different LM methods, provided that the pretrained model $f_{\\rm pre}$, input $x$, and input transformations $f_{\\rm in}$ are the same across different LM methods (details in Appendix Sec. C.1).\n- The expected accuracy calculation is based on conditional probabilities where $x$ is implicitly part of the condition. As all LM methods operate on the same $x$, we can safely omit it, allowing us to concisely highlight the differences of LM themselves.\n\nWe will also add a note in the main text to clarify this omission, emphasizing our focus on comparing LM methods and the scope of this analysis.\n\n**W5:** Thank you for your question. We will clarify it in the next version. $y_r^{\\rm S}$ means a class that is more relevant to $y_{i}^{\\rm T}$ (line 256), while $y_{\\bar r}^{\\rm S}$ is a class that is less relevant to $y_{i}^{\\rm T}$ (line 257). Here we assume two classes, $y_{r}^{\\rm S}$ and $y_{\\bar r}^{\\rm S}$, in the output space of the pre-trained model, which satisfy $p(Y^{\\rm T}=y_i^{\\rm T}|Y^{\\rm S}=y_r^{\\rm S}, X^{\\rm T})>p(Y^{\\rm T}=y_i^{\\rm T}|Y^{\\rm S}=y_{\\bar r}^{\\rm S}, X^{\\rm T})$.\n\n**W6:** Thank you for your question. Please refer to our reply to **Common Question 1**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wLKXAthYZm", "reviewer": "Reviewer_iW2K", "summary": "This paper focuses on a technique called label mapping (LM) which is used in visual reprogramming. It finds the relation between pre-trained labels and downstream labels. In conventional fine-tuning, LM is just the backpropogation using the loss function. However, this paper and previous similar papers aim to develop a gradient-free way to find the weights in the last layer in the conventional fine-tuning. The whole paper is based on Bayes' rules (using it on the output of a pre-trained model, which is interesting), which is solid and theoretically justified.", "questions": "See the weaknesses above.", "limitations": "No concerns for this paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Good paper flow. The flow of this paper is good and appreciated. Directly analyzing the loss using the Bayes' rule is clear.\n\n2. Extensive experiments. Except for the good performance, the authors also try to understand the good performance, which is interesting. Especially on the visualization of top relationship between pre-trained labels and downstream labels. The result is convincing, as the selected top pre-trained labels are indeed related to downstream tasks.\n\n3. Timing topic. I can see some applications which might need this kind of techniques. Keeping the integrity of a pre-trained model has many advantages, e.g., not lose the generalization of the pre-trained model.\n\n4. Solid analysis. Two theoretical contributions are included in this paper. One is how to embed a pre-trained model into p(y^T|x^T), another one is how BLM has a higher accuracy.", "weaknesses": "1. Benchmark datasets follow the previous papers, which is good. However, these tasks seem not the most relevant to what you propose in this paper. Are there some scenarios where the fine-tuning the last layer is not feasible? \n\n2. The paragraph between 253-266 is confusing, which can be moved to other section. Appendix perhaps.\n\n3. One major drawback of the presentation is the algorithm tables. The equations are still not the best way for practitioners. Main algorithm table should be moved in Section 4. After looking at the algorithms, it is easy to find that the proposed method is easy to be implemented. However, I cannot feel that after I read the section 4.\n\n4. How to choose Padding or Watermarking in practice?\n\n5. Experiments to increase n are not necessary. Could the authors explain why it matters? As long as the method is valid, the accuracy will be improved when n increase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** Thank you for your question. This paper falls in the scope of Visual Reprogramming (VR). Thus, the experimental setting of this paper aligns with those used in previous VR studies, particularly ILM [1], to ensure comparability and consistency in the field. This setting serves as an established benchmark for evaluating VR techniques.\n\nRegarding when fine-tuning the last layer is not feasible, we consider the following cases:\n1. copyright and legal constraints: modifying some pretrained models may violate licensing agreements or intellectual property rights.\n2. prevent catastrophic forgetting: keeping the pretrained model intact helps to maintain its general knowledge across tasks\n3. black-box optimization: in addition, our method is useful when only the predicted logits of input are available. It does not need access to the model's internal parameters which fine-tuning clearly struggles to handle.\n\n**W2:** Thank you for your question. Due to space limitations, lines 253-266 of the original text are not fully described. We will move this section to the appendix in the next version and include detailed information.\n\n**W3:** We appreciate your feedback regarding the presentation of Sec. 4 and algorithm tables. We acknowledge your point about the placement of the algorithm tables. Due to space constraints, we had initially placed them in the Appendix (lines 455 and 462). However, we recognize the importance of making these algorithms more readily accessible to readers. We plan to move the main algorithm table to Sec. 4 as you suggested in the final version.\n\nWe understand your concern regarding the equations in Sec. 4. While the mathematical formulations are important for a rigorous presentation, we will smooth out the section with more practical insights. We aim to strike a balance between mathematical precision and practical applicability.\n\n**W4:** Thank you for your question. The use of Padding or Watermarking depends on the specific downstream tasks, and we believe the selection may partly depend on the relationship between the pretrained model's input size and the downstream task's image dimensions.\nFor instance, let’s consider the results of BLM+ using ResNet-18 as the pre-trained model:\n\n| | GTSRB | SVHN | CIFAR10 | CIFAR100 | Flowers102 | EuroSAT | OxfordPets | SUN397 | UCF101 | Food101 | DTD |\n| -------- | -------- | -------- | -------- | -------- | -------- | ------- | -------- | -------- | -------- | -------- | -------- |\n| Image Size | 32 | 32 | 32 | 32 | 128 | 128 | 128 | 128 | 128 | 128 | 128 |\n| Padding | 54.3 | 74.2 | 66.8 | 30.6 | 50.1 | 86.7 | 70.6 | 18.7 | 32.0 | 25.1 | 43.9 |\n| Watermarking | 82.0 | 78.8 | 75.7 | 41.6 | 44.1 | 84.8 | 73.3 | 19.4 | 35.4 | 22.9 | 43.0 |\n\nOur observations:\n- watermarking performs better when there is a significant size disparity: for tasks with much smaller images (32x32, e.g., GTSRB, SVHN, CIFAR10, CIFAR100) compared to the pre-trained model's input (224x224), watermarking often outperforms padding. It prevents introducing too many parameters around the image and significantly downscaling it.\n- padding may perform better with larger downstream images in some cases: for tasks using larger images (128x128, e.g., Flowers102, EuroSAT), padding tends to perform better. It maintains the original image integrity by avoiding pixel value alterations.\n\nIn general, the choice impacts how we adapt pretrained model to new tasks. Therefore, the goal for input VR is to maximize the transfer of learned features while accommodating the new task's visual characteristics - the optimal choice may depend on factors beyond just image size and deserves future explorations.\n\n**W5:** Thank you! Indeed, the accuracy typically improves with larger training sets. Our focus on smaller $n$ values serves as a purpose in evaluating the robustness of our proposed methods, BLM/BLM+, compared to the baseline ILM under limited data availability conditions. Our rationale for this evaluation is:\n1. Practical implications: VR was proposed for adapting pretrained model to data-limited tasks. In this sense, obtaining large-scale labeled downstream task data can be challenging. By demonstrating robust performance with smaller $n$ (25%, 50%, and 75% of the original size), we highlight the practical advantages of BLM/BLM+ in data-limited scenarios, which better aligns with real-world applications.\n2. Overfitting risk: Smaller training sets inherently carry a higher risk of overfitting. By testing our methods under these conditions, we can better assess their generalization capabilities compared to the baseline ILM.\nThus, these experiments were conducted to validate the effectiveness and reliability of BLM/BLM+ across various data regimes. While increasing $n$ may not be that meaningful, our evaluation aims to demonstrate that BLM/BLM+ maintains competitive performance even with limited data, thus providing a more comprehensive evaluation of the practical utility.\n\n[1] Chen et al. Understanding and improving visual prompting: A label-mapping perspective. In CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Common Question 1:** Concerns about the required number of epochs and training time for BLM/BLM+

**Response 1:** Regarding the number of epochs, we initially used 200 epochs as with the original papers to ensure a fair comparison with the baseline methods. However, during the rebuttal stage, we conducted additional experiments to assess the impact of different epoch numbers (60, 100, 150) on our BLM/BLM+ model, using ResNet-18 as the pretrained model. The results are shown in Table 1.
**Table 1. Epoch Numbers and Testing Accuracy of Different Methods**
|    |    |BLM| (ours)|    |  |  BLM+    | (ours) |      || ILM  | FLM  |
|---------------------|------|------|------|------|--------------|------|------|------|---|------|------|
| Epochs              | 60          | 100  | 150  | 200  | 60           | 100  | 150  | 200  || 200  | 200  |
| Average on 12 Tasks (%) | 44.5        | 45.2 | 45.5 | 45.3 | 45.8         | 46.4 | 46.9 | 46.7 || 40.6 | 37.2 |

We found that running 100 epochs yields results comparable to those achieved with 200 epochs. This demonstrates that BLM and BLM+ require **less convergence time**, highlighting their efficiency.

Training time for one epoch is listed in Table 6 of our paper submission. Additionally, the total training time for one task, in comparison with baseline visual reprogramming methods and fine-tuning methods, is calculated below:

**Table 2. Time Consumption of Different Methods on Flowers102 Dataset (Single A100 GPU)**
|             |                  | VR: Baselines |       |    VR: Ours   |                |      VR: Ours             |                    |   Finetuning Methods  |                  |
|-------------|------------------|:-------------:|:-----:|:-----------------:|:------------------:|:-----------------:|:------------------:|:---------------------:|:----------------:|
|             |                  | FLM           | ILM   | BLM（200 epochs） | BLM+（200 epochs） | **BLM（100 epochs)** | **BLM+（100 epochs)** | Finetuning Last Layer | Fully Finetuning |
|  ResNet-18  | Parameter Number | 0.10M         | 0.10M | 0.10M             | 0.10M              | 0.10M             | 0.10M              | 0.51M                 | 11.7M            |
|             | Training Time (min) | 11.97         | 12.04 | 11.95             | 13.06              | **6.03**              | **6.52**               | 14.03                 | 15.28            |
| ResNeXt-101 | Parameter Number | 0.10M         | 0.10M | 0.10M             | 0.10M              | 0.10M             | 0.10M              | 2.0M                  | 88.8M            |
|             | Training Time (min) | 24.68         | 24.81 | 24.51             | 24.71              | **12.33**             | **12.44**              | 24.49                | 35.07            |

Combined with the results in the table, we analyze the efficiency of BLM and BLM+ from three perspectives:

(1) **Extra Consumption of Calculating Mapping Matrix $\omega$ Compared with One-to-One Mapping:** Compared to the baseline method ILM, the additional cost for BLM/BLM+ primarily involves the gradient-free multiplication and division within the mapping matrix (which is sized according to the source and target label spaces, 1000 × 102 in this case). This additional cost is minimal, as shown by the 4th-6th columns in Table 2.

(2) **Time Consumption of Updating Mapping Matrix $\omega$ per Epoch:** The baseline method FLM calculates the mapping $\omega$ once and keeps it fixed, while ILM and our methods update $\omega$ at each step. However, updating $\omega$ does not require running the model to obtain current predictions each epoch. Instead, predictions from the most recent epoch can be reused. As a result, there is no noticeable time overhead for updating $\omega$ per epoch, as indicated by the 3rd-6th columns in Table 2.

(3) **Time Consumption Compared with Finetuning Methods:** Since BLM/BLM+ use **fewer parameters** and **converge in fewer epochs**, they are significantly faster than finetuning the last layer or the entire model. This is demonstrated in the 7th-10th columns of Table 2.

---

**Common Question 2:** Detailed Explanations of Conditional Probabilities in Section 4.1

**Response 2:** Eq. (6) and (7) aim to estimate $p(Y^{\rm T} = y^{\rm T}, Y^{\rm S} = y^{\rm S} \mid X^{\rm T})$ and $p(Y^{\rm S} = y^{\rm S} \mid X^{\rm T})$ respectively. Here, $X^{\rm T} \in \mathcal{X}^{\rm T}$ represents a variable from the downstream task input space, while $Y^{\rm T} \in \mathcal{Y}^{\rm T}$ and $Y^{\rm S} \in \mathcal{Y}^{\rm S}$ are variables from the target and source label spaces, respectively.

The conditional probability $p(Y^{\rm T} = y^{\rm T}, Y^{\rm S} = y^{\rm S} \mid X^{\rm T})$ represents the joint distribution of $Y^{\rm T}$ and $Y^{\rm S}$, given the input reprogramming $f_{\rm in}$, the pretrained model $f_{\rm pre}$, and the variable $X^{\rm T}$ of the downstream task. Similarly, $p(Y^{\rm S} = y^{\rm S} \mid X^{\rm T})$ represents the distribution of $Y^{\rm S}$ under these conditions.

For example, consider the following setup:
- $\mathcal{Y}^{\rm T} = \lbrace\tt Cat, \tt Dog\rbrace$,
- $\mathcal{Y}^{\rm S} = \lbrace\tt CockerSpaniel, \tt EnglishSpringer, \tt EgyptianCat\rbrace$,
- Downstream samples are $ \lbrace(x_1, {\tt Dog}), (x_2, {\tt Dog}), (x_3, {\tt Dog}), (x_4, {\tt Cat})\rbrace$.

If the reprogrammed predictions calculated by $f_{\rm pre}(f_{\rm in}(x_i \mid \theta))$ are $\lbrace x_1: {\tt CockerSpaniel}, x_2: {\tt CockerSpaniel}, x_3: {\tt EnglishSpringer}, x_4: {\tt EgyptianCat}\rbrace$, then $p(Y^{\rm T} = y^{\rm T}, Y^{\rm S} = y^{\rm S} \mid X^{\rm T})$ can be estimated as a 2 \* 3 matrix with the following nonzero values:
- $p(Y^{\rm T} = {\tt Dog}, Y^{\rm S} = {\tt CockerSpaniel} \mid X^{\rm T}) = \frac{1}{2}$,
- $p(Y^{\rm T} = {\tt Dog}, Y^{\rm S} = {\tt EnglishSpringer} \mid X^{\rm T}) = \frac{1}{4}$,
- $p(Y^{\rm T} = {\tt Cat}, Y^{\rm S} = {\tt EgyptianCat} \mid X^{\rm T}) = \frac{1}{4}$.

Meta-review: This paper presents work on visual reprogramming -- leveraging existing vision models and adapting them to solve tasks involving different labels.  The core contribution is a Bayesian approach to label mapping that alleviates shortcomings of previous (one-to-one mapping) approaches, and better utilizes cross-category information in conducting label mapping.

The reviewers noted the clear strengths of the paper in this regard, with a clear motivation, solid execution, and impressive empirical results.  A few minor issues were raised in the reviews, which were largely addressed in the author responses.

Visual reprogramming, and more broadly adapting pre-trained models to solve new tasks, is an active, important area of research.  This paper makes solid contributions to this literature, and as such is recommended for acceptance to NeurIPS.
Decision: Accept (oral)