Endowing Visual Reprogramming with Adversarial Robustness
Authors: Shengjie Zhou
Xin Cheng
Haiyang Xu
Ming Yan
Tao Xiang
Feng Liu
Lei Feng
Venue: ICLR 2025 Poster (2025)
Keywords: visual reprogramming
adversarial robustness
risk bound
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Visual reprogramming (VR) leverages well-developed pre-trained models (e.g., a pre-trained classifier on ImageNet) to tackle target tasks (e.g., a traffic sign recognition task), without the need for training from scratch. Despite the effectiveness of previous VR methods, all of them did not consider the adversarial robustness of reprogrammed models against adversarial attacks, which could lead to unpredictable problems in safety-crucial target tasks. In this paper, we empirically find that reprogramming pre-trained models with adversarial robustness and incorporating adversarial samples from the target task during reprogramming can both improve the adversarial robustness of reprogrammed models. Furthermore, we propose a theoretically guaranteed adversarial robustness risk upper bound for VR, which validates our empirical findings and could provide a theoretical foundation for future research. Extensive experiments demonstrate that by adopting the strategies revealed in our empirical findings, the adversarial robustness of reprogrammed models can be enhanced.

Published as a conference paper at ICLR 2025
ENDOWING VISUAL REPROGRAMMING WITH ADVER- SARIAL ROBUSTNESS
Shengjie Zhou' l+ Xin Cheng Haiyang Xu²
Ming Yan
Tao Xiang' Feng Liu
Lei Feng
Chhnngiing University 2 Aliiaba Group
University of Melbourne Southeast University "Idealism Technology (Beijing)
ABSTRACT
Visual reprogramming (VR) leverages well-developed pre-trained models (e.g.. a pre-trained classifier on ImageNet) to tackle target tasks (e.g., traffic sign recog- nition task), without the need for training from scratch, Despite the effectiveness of previous VR methods, all of them did not consider the adversarial robustness of reprogrammed models against adversarial attacks, which could lead to unpre- dictable problems in safety-crucial target tasks. In this paper, empirically find that reprogramming pre-trained models with adversarial robustness and incorpo- rating adversarial samples from the target task during reprogramming can both improve the adversarial robusiness of reprogrammed models. Furthermore, we propose theoretically guaranteed adversarial robustness risk upper bound for VR, which validates our empirical findings and could provide a theoretical foun- dation for future research. Extensive experiments demonstrate that by adopting the strategies revealed in our empirical findings, the adversarial robustness of re- programmed models can be enhanced.
INTRODUCTION
Visual pre-trained models (Krizhevsky et al. 2012 Russakovsky al.) 2015 Huang et al.) 2017 Radford et al. (202T) obtained from the (usually large) source domain serve as a powerful foundation for a variety of visual target tasks. When faced with the relevant downstream tasks, the pre-trained models can be adapted to solve the tasks through fine-tuning. However, a significant limitation of
fine-tuning methods (Pan et al. 201T Ghifary et al. 2014, Kumar et al. 2022) is their requirement for partial or complete modification of model parameters, resulting in high training and storage costs, particularly when using with large models. In contrast, visual reprogramming (VR) (Lee al. 2020
Kloberdanz et al. 2021 Chen et al.) 2021] Neekhara et al. 2022 Wang et al. 2022: Chen 2022 provides an alternative and efficient way re-purposing the well-developed pre-trained models from source domains. Specifically, VR does not require modifications to the parameters of the pre-trained model. Instead, as illustrated in Figure T it fixes the pre-trained model and incorporates an input transformation (Yang et al. 2021 Bahng al. 2022 Cai al. 2024 Tsao et al.. 2024] and an output label mapping (Elsayed et al. 2018 sal et al. 2020 Chen et al., (0236) at the input and output stages, respectively, adapt the requirements of the target domain. Moreover, compared with general prompting methods (Jia et al. 2022] Cheng et al. 2023) in visual tasks, VR offers a model-agnostic approach that preserves the visual essence of the input images (Cai et al. 2024).
Previous research has proposed numerous VR methods, which have achieved satisfactory results across a wide range of tasks, demonstrating the considerable potential of VR.
Despite many previous studies on VR, they have not considered the adversarial robustness (Szegedy) 2013 Ashmore al. (202T) of the reprogrammed models in target domains, which is an important standard for measuring the quality of learned models, especially in safety-crucial domains like med-
ical diagnosis (Hamid et al. 2017 Kadampur & Al Riyaee 2020), autonomous driving (Grigorescu
et al. 2020
Feng et al. 2021), and criminal justice (Zhong et al. 2018 Chalkidis et al. 2019).
"These authors contributed equally to this work.
1 'Correponning author.

---

Published as a conference paper at ICLR 2025
Trainable Moduues
Frozen Modules
Adversarial Robustnnss Modulis
Visual Input Transformation
trained Models
Output Label Mapping
RandLM
ResNett18
ADXL Crra
Frequ
FullyL
ResNet-S5
IterLM
Padding
Reszing
gdouse T papee
Figure 1: Overview and key highlights of VR with adversarial robustness. The main components Training Data, which comes from supervised data of the target domain task, including the cleaned data and generated adversarial data; Visual Input Transformation. which transforms data from the target domain to the inputs of the pre-trained model, including both Padding and Resizing methods; Pre-trained Models, which are well-developed classifiers in the source domain, including both naturally trained models (N.T.) that lack adversarial robustness and adversarially trained mod- els (A.T.) that possess adversarial robustness; and Output Label Mapping, which maps labels from the source domain to the target domain, including one-to-one mapping (e.g.. M,, FreqI.M and IterLM) and fully mapping (FullyLM). We indicate that selecting adversarial data for reprogram- ming and choosing ad adversarially pre-trained models can both improve the adversarial robustness of the reprogrammed model.
are:
Adversarial robustness refers to the ability of a model to maintain its performance under adversar- ial attacks (Goodfellow et al. 2014 Carlini & Wagner 2017; Yao et al. 2019) which aim to add small perturbations that cause incorrect predictions. Numerous studies (Szegedy) 2013 Biggio et al.] 2013) have demonstrated that naturally trained models exhibit significant vulnerabilities when fronted with adversarial attacks. However, it remains unclear whether the same phenomenon occurs in VR. Unfortunately,
shown in Figure 2 we empirically find that the reprogramming of natu- rally pre-trained models under various VR results in virtually no adversarial robustness on GTSRB. Notably, C-AVP (Chen al.] (203a)) focuses on improving the robustness of naturally pre-tree trained models within the same task, whereas it does not investigate the adaptation of a pre-trained model to achieve robustness across distinct tasks. Therefore, ensuring adversarial robustness essential for the effective development of VR.
In this work, we for the first time investigate the issue of adversarial robustness within the context of VR. Our empirical findings reveal that while reprogramming naturally pre-trained model can achieve high accuracy, its performance on adversarial examples is nearly zero. In contrast, repro- gramming an adversarially pre-traines mre-trained model yields certain degree of adversarial robustness in the target domain, indicating that current reprogramming methods can partially leverage the adversarial robustness of the adversarially pre-trained models. Inspired by adversarial training (e.g.. PGD-AT (Madry et al. 2018)) and TRADES (Zhang et al.) 2019a)), we incorporate adversarial examples from the larget domain during the reprogramming process, which enhances the adversarial robustness of reprogrammed models, particularly yielding significant improvements when using an adversarially pre-trained model. For example, in Figure 2 the adversarial robustness of either reprogramming an adversarially pre-trained ResNet-50 with natural reprogramming or reprogramming naturally pre- trained ResNet-50 with adversarial reprogramming is higher than the original, and the adversarial adversarially pre-trained ResNet-50 with adversarial reprogram-
robustness of reprogramming ming is even better. These strategies are illustrated Figure
Furthermore, in order to theoretically understand the adversarial robustness of VR, we present a the- oretically guaranteed adversarial robustness risk upper bound for VR. Specifically, our theoretical analysis establishes a connection between the target domain and the source domain, demonstrat- ing that the adversarial robustness of VR is primarily influenced by these factors: 1) the adversarial robustness of the pre-trained model in the source domain; 2) the adversarial margin disparity discrep- ancy between the source and target domains; 3) the adversarial marginal risk in the target domain for the optimal model with adversarial robustness of source domain. This indicates that replacing
2

---

Published as a conference paper at ICLR 2025
pre-trained model lacking adversarial robustness with one that possesses adversarial robustness, as well as incorporating adversarial examples from target domain during reprogramming, can result in reduced upper bound of adversarial risk, thereby validating findings. Our main contributions can be summarized as follows:
Our main contributions be summarized as follows:
This study is the first explore adversarial robustness within the context of VR, demonstrating the vulnerability of existing VR methods in the face of adversarial attacks.
Our empirical results indicate that reprogramming pre-trained models with adversarial robustness and incorporating adversarial examples from the target task during the reprogramming both improve the adversarial robustness of VR.
We provide an adversarial risk upper bound for VR, which validates our empirical findings and provides a theoretical foundation for future research.
2
PRELIMINARIES
In this section, we introduce preliminary knowledge of the adversarial training and the visual repro- gramming, well as the notations we used throughout this paper.
2.1 ADVERSARIAL TRAINING
Natural Training (N.T.). Let us consider k-class classification task with the input data (x, y) R x [k] sampled from an unknown data distribution with probability density p(æ, y). Most existing learning algorithms for such classification tasks optimize the natural objective loss ENT (e.g.. cross- entropy loss) on clean examples to derive a well performing classifier f Rd + R² with parameters 0. The goal of N.T. can be expressed
min Fo(aw) (CNT (f 0).
However, previous studies (Szegedy 2013 Biggio et al. 2013) have demonstrated that despite the high clean accuracy of naturally trained models on unperturbed examples, their accuracy signifi- cantly deteriorates when exposed to examples that have been subtly perturbed by adversaries (i.e., adversarial examples). These perturbations are so small that they are invisible to the human eye. We refer to the performance on adversarial examples as adversarial robustness. For simplicity, we denote adversarial robustness by robustness in the following.
Adversarial Training (A.T.). In contrast N.T., A.T. aims to improve the robustness of model by incorporating adversarial examples during the training. The goal of A.T. can be expressed as min-max optimization process:
min E p(a.w) max ENT (f(x + 10),)
[l
where 7 is the adversarial perturbation, c is the adversarial perturbation radius and ||||| represents the norm operation (i.e., -norm).. Here, the inner layer attempts find the adversarial examples that causes the classifier to misclassify, while the outer layer focuses on optimizing the model parameters accurately elassify these adversarial examples.
2.2 VISUAL REPROGRAMMING
Let Xss C Rd be the ds di-dimensional source domain space and V's = f1...., ks} be the source domain label space where ks denotes the number of source domain classes. Assume we have pre-trained model [s Xs + Ris with parameters Oş trained on the source domain dataset Dş {(x5 v | where each source domain example (a. ufr) E X's x 3/s is assumed to be sampled from an unknown source data distribution with probability density p(as, Us). It noteworthy that fs is frozen (i.c., Os is non-trainable) in VR, we will omit Os in following parts. For the target domain of VR, we denote the dimeenssion target domain space as XT C gdr, and the target label space Dr (1...., kr) where by denotes the number of target domain classes.
Let DT
NT be the target domain training set, where each target domain example (T] ya )) i=
3

---

Published as a conference paper at ICLR 2025
(a .9!) Xy x yr is assumed to be sampled from unknown data distribution with probability density p(pT,91) UT). Then, the training objective of natural visual reprogramming is:
min Eptay ur) | ((four o fs o (aT) Oin, Ounr) (onn),yT)]
(3)
Ou.
where fim : Ry + Xs with parameters Om is the visual input transformation, fou R - RAT with parameters our the output label mapping, and < RCT x Yrr + R is the natural loss function of the target domain. It is worth noting that VR is to reprogram a pre-trained model fs from a complex source domain to a simpler target domain, so VR assumes (1) the dimension of the target data is not greater than that of the source data (i.c., dy dg); (2) the number of target class labels is not greater than that of the source class labels (i.c., ky S kg) (Chen 2022 Cai et al. 2024).
Visual Input Transformation. Due to the discrepancies between the input spaces of the target domain and the source domain, data from the target domain cannot be directly used as input for the source domain model. Consequently, it is necessary to transform the data from the target domain. This approach primarily includes Padding- -bared and Resizing-baat -based methods. The madding-based method (Pad) (Tsai al., 2020; Chen et al.) 2023b) adds learnable parameter 5 around the image from the target domain to ensure the resulting image conforms to the input shape required by the source model. The Resizing- based method, which includes fully-watermarking (Full) (Bahng et al. 2022) and sample-specific multi-channel masks (SMM) (Cai et al. 2024), first resizes the target domain image to the input shape of the source model using interpolation algorithms. Then, the learnable parameter 5 is multiplied by a mask M and added to the resized image. The difference between the two lies in that the mask M in watermarking is a manually set, fixed matrix containing only Os and 1s, while in SMM, M is generated by learnable lightweight network that takes the resized image as input and produces an input-dependent mask. Based on the above, we can formalize visual input transformation as:
fin 0i = M)) = g(x) + + M,
(4)
where g(-) represents the transformation applied to the target input (e.g.. padding or resizing).
Output Label Mapping, The output label mapping can be categorized into non-iterative and it- erative methods. Non-iterative methods include one-to-one Random Label Mapping (RandLM) resquency-basee Label Mapping (FregLM) (Tsai al. 2020). RandLM establishes a random mapping between the output space of the source domain and that of the target domain at the beginning of training. FreqL.M matches the labels of the source and target domains based on the prediction frequency of the source model on g(æ) at the beginning of train- ing. Iterative methods include one-to-one Iterative Label Mapping (IterLM) (Chen et al., 2023b) and Fully Label Mapping (FullyLM). IterLM iis an iterative version of the FreqLM method. Specif- ically, IterLM recalculates the mapping relationship using FreqLM during each round of training. FullyLM employs fully connected layer and updates its parameters using gradient descent. These output label mapping methods can be uniformly represented as:
(Elsayed et al., 2018) and one-to-one
fowt(x fin fs, Pout W) W fm 0m) b.
(5)
For the one-to-one mapping methods, W {0, 1] x a matrix with at most one element equal to in each row (i.e.. Σド= W.g = 1), and b is an all-zero vector (i.e., b = (T) ). For FullyLM, W and are learnable parameters (i.e., W E RT and b E IR*T).
VISUAL REPROGRAMMING WITH ADVERSARIAL ROBUSTNESS
We define the label function hy : x + arg max, [=()] where f is a multi-class score model (e.g., fs and fout fs o Sin) and [f(=:)] denotes the i-th score output by the model. Then, for given hy, the expected adversarial risk with respect to the distribution If is defined as:
Rid (hy,e) A (mag)uu max 1[ (x + 7) * vl.
(6)
lhlls
where [[-] denotes the indicator function. The adversarial aims for VR is to learn visual input transformation fin and output label mapping fout that tries to minimize adversarial risk Rej (fout o Is fin; e) in target domain T.
4

---

Published as a conference paper at ICLR 2025
25
Natural Training
Adversarial Training
Robustness (%)
20
20.12
32.71
17,27
15
15.12
12.70
10
9.74
1,67
0
0.03
RN18 (N.T.)
RN50 (N.T.)
XCIT-S12 (N.T.) RN18 (A.T.)
RN50 (A.T.) XCIT-S12 (A.T.)
Pre-trained Model
Figure 2: The test performance of three different pre-trained models (e.g.. ResNet18, ResNet50, and XCTT-S12, all pre-trained on ImageNet-1K) for reprogramming on the GTSRB dataset. There are two versions for each pre-trainee model, including naturally trained models (N.T.) that lack adversarial robustness and adversarially trained models (A.T.) that possess robustness. The visual input transformation is fixed as Pad and the output label mapping is fixed as FullyLM.
Inspired by the effective reuse well-developed pre-trained models in VR, one may apply Eq. B reprogram adversarially pre-trained model of the source domain that possesses satisfac- tory robustness, such as an adversarially trained (A.T.) model, in order to repurpose its robustness. Furthermore, since adversarial training known effectively enhance robustness for model, in- corporating adversarial examples of target domain into the reprogramming process is anticipated improve robustness on the target domain, even the pre-trained model such as naturally trained (N.T.) model. Moreover, simultaneously using a pre-trained model with robustness and incorporating adversarial examples during reprogramming could potentially yield further robustness of the reprogrammed model in the target domain.
inherently robust,
To implement adversarial training, we integrate adversarial examples directly into the VR training objective as follows:
min (p(erer) max Kr (Vour Is (an(( +- 0 o UT) Uhilse
where (0.. 0 are the trainable parameters (i.e., visual input transformation and output label mapping) of VR. Furthermore, inspired by TRADES Zhang al. (2019a), add robustness regularization term to the VR training objective, which measures the magnitude the output differ- ences between the original examples (æT,y) v) and the adversarial examples (æT + 7) Thus, this robust training objective can be expressed as follows:
min E Ep(ar uT) LeNT ((Sor Ss fin) Om, 0omt) .IT)
(8)
+ A max lxL ((fo fs fin((T 0i, o I(ffut o fs fin) + y 0m, Q
where À 2 0 is hyperparameter controlling the trade-off between natural loss and robust regular- P(t)
denotes the KL divergence which is used measure the distance between the two probability distributions. In Figure 2 we show the results of repro- gramming using adversarial examples generated from the target domain and the results obtained from reprogramming using adversarially pre-trained models.
ization, and (KL (PIlQ) = E. log
Q
THEORETICAL UPPER BOUND
As previously discussed, leveraging adversarially pre-trained models for reprogramming and incor- porating adversarial examples from the target domain may both enhance the robustness of repro- grammed models. this section, we provide the first adversarial risk upper bound for VR, which validates findings and provides theoretical guarantee for future research.
5

---

Published as a conference paper at ICLR 2025
4.1 ADVERSARIAL MARGIN LOss
Due to the significant role of margin theory in ensuring generalizability. inspired by (Zhang et al. 2019b), employ margin loss as substitute for the 0-1 loss. The margin of model I the example (ær, y) is defined as:
P, v) (1(())), max(f())lv))
(9)
v'
Then, the corresponding adversarial margin risk can be represented as follows:
Radx, (f) é B (a Ru
max D (py(< + 7,,)))
(10)
Ubils
where (p(-) is a piecewise function defined as follows:
P p
(p(2)
1 z/p
0 2
(11)
=
Lemma 1. Given distribution U and socre model f. for any p O. adversarial risk Ryd (hy) and adversarial margin risk Ru Radv.(p) (f) have the following lemma:
Ry
adv (p)
(f) Rud (hy).
The proof of Lemma is provided in Appendix A.1 Lemmal Tindicates that the adversarial margin risk can serve upper bound for the adversarial risk. Based this relationship. can derive the adversarial risk upper bound for VR through the margin loss.
4.2 ADVERSARIAL MARGIN DISPARITY DISCREPANCY
Margin Disparity Discrepancy (MDD) aims quantify the inconsistency between domains by com- paring the marginal performance differences of classifier across two distinct domains. However, MDD does effectively quantifies the differences in adversarial marginal performance of the classifier across the domains. Therefore, we propose the definition of Adversarial Margin Dispar- ity Discrepancy of Visual Reprogramming with Adversarial Robustness (AMDD- (RRA within the framework of robustness. Before that, we first define the adversarial margin disparity (AMD) be- tween f' and f on domain 21 as:
dispa (p) (P' (f' Eapp(a) A
max ( + (x))))))++)))) ()11
(12)
Uivise
AMD quantifies the maximum margin disparity between models f' and f within the e-perturbation range around the instance æ, meaning the expectation of inconsistent classification between I' and under e-perturbation in the domain 24 Based on AMD, we introduce the definition of AMDD- MRA.
Definition (AMDD-VRA) Suppose that Fs is the hypothesis spaces of the source domain model. For a given pre-truined Is from source domain, visual input transformation fin and output label mapping four the AMDD-VER VRA between source domain and target domain T is defined as (o)v.(o) Sonofso (S,
disp adv.(o) (fout o f's fins fout fs ffn) ddssps adv.(o) ( (fs:
sup
From Definition 1, AMDD-VIVVV VRA measures the adversarial distribution distances between the source domain S and the target domain T. It can be observed that fi and fou influence the value of AMDD- VRA. For instance, when the source domain and target domain are completely identical, both fm and four are identity mappings, the AMDD-VRA value is O. However, in the context of VR settings, where the source and target domains are not identical, fin and four can play a role in either narrowing or widening the adversarial discrepancy between the two different domains. Furthermore, based on the definition of AMD, can be inferred that whether adversarial training is employed for fin and fout may impact the first term of AMDD-VER thereby affecting the overall value of AMDD-VRA. From the definition of AMDD- VRA and Lemma I we can induce the following adversarial risk upper bound for VR:

---

Published as a conference paper at ICLR 2025
Theorem 1. Suppose that the f3 arg min(Ri (Rsiv. (Js)3 is the optimal classifier with robustness
fsEFs
in source domain. For given output label mapping fout, pre-trained source domain model f's, and visual input transformation fin. the following adversarial risk upper bound for VR from source domain S to target domain T holds:
Raph (he ofo0 Re (fs) + d godv.(p) fo (S,T) + R2 (Fout o fi)) X,
Rg
where x= = RR Radv.(0) (f') is a constant associated with the source domain.
The proof of Theorem is provided in Appendix A.2 Theorem reveals that the adversarial risk of VR is influenced by the following factors: 1) the adversarial margin risk of the selected pre- rained model fs in the source domain; 2) the AMDDD-RA between the source domain and the target domain; 3) the adversarial marginal risk in the target domain for the optimal adversarial trained source domain model f3 after reprogramming: 4) the adversarial margin risk of the optimal adversarial trained source domain model / in the source domain, which be considered constant value X. This theory validates our previous findings. In the next section, we will provide more experiments demonstrate the effectiveness of our strategy.
EXPERIMENTS
In this section, we empirically demonstrate that by adopting the strategies revealed in our findings, the robustness of reprogrammed models can be enhanced.
5.1 EXPERIMENTAL SETUPS
Pre-trained Source Models and Target Tasks. We select ImageNet-1K (Russakovsky al. 2015)
(the most commonly used subset of the well-known ImageNet image classification dataset) dataset as the source task. This dataset encompasses 1,000 object classes and contains 1,281, 167 training im- ages. The source domain pre-trained models fs include ResNet-18 (RN18), ResNet-50 (RN50) (He) al. (016)) and XCTT-S12 (EI-Nouby et al., (02T), all of which have an input shape of 224 x 224. The weights of the naturally pre-trained models (ResNet-18 (N.T.) and ResNet-50 (N.T.)) are ob- tained from the official PyTorch model repository, while the weights of the adversarially pre-trained models (ResNet-18 18 (A.T.), ResNet-50 (A.T.) (Salman al.) 2020) and XCTT-S12 (Debenedetti et al.) ) are obtained from Robustbench (Croce et al. 2021). For the target datasets, we selected 10, CIFAR- 100 (Krizhevsky et 2009),, and GTSRB (Stallkamp al. 2012), where the target classes are 10, 100, and 43. Detailed information about all datasets can be found in Appendix
Our goal is obtain robust classifier via VR.
Implementation. In the implementation of VR, we fix the pre-trained model fs and select the visual input transformation model fm from Pad, Full, and SMM, while the output label mapping fout is chosen from RandLM, IterLM, and FullyLM. Additionally, during the implementation of adversarial training and testing, we generate adversarial examples using 10-step Projected Gradient Descent (Madry et al. 2018) (PGD-10) for training, setting the perturbation radius to = 4/255 under Loo- norm, consistent with the settings of adversarially pre-trained models. When using TRADES for adversarial training of VR, we set X = 6, which is a commonly used setting in previous adversarial training research. Moreover, we use the AdamW optimizer to train all methods for 60 epochs, and repeat the samplin--and-traiing process 3 times. The learning rate of the optimizer is searched from set (1x 10-3 3 5x 10 10-4), while the weight decay of the optimizer is searched from set (10-2,10* 3).
Details of the experiment, complete results, and some visual results can be found in Appendix B
5.2 EXPERIMENTAL RESULTS
The experiments are conducted on the combinations of pre-trained models and training methods, encompassing four distinct cases: Strategy 1: VR with a naturally pre-trained XCIT-S12 and us- ing natural training: Strategy 2: VR with naturally pre-trained XCIT-S12 and using adversarial training, Strategy 3: VR with an adversarially pre-trained XCIT-S12 and using natural training; Strategy 4: VR with adversarially pre-trained XCIT-$12 and using adversarial training. Table
7

---

Published as a conference paper at ICLR 2025
Table 1: The performance (mean (%) - std (%) comparison of VR with XCIT-SI2 on CIFARIO,
where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustmess (clean accuracy), where the std of clean accuracy omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model / VR using Adversarial Training
In Out Out
Strategy 1 (x / x)
Strategy 2 (X v/)
Strategy 3 (V/ x))
Strategy 4 (V/ /V)
Pad/ / RandLM
0.06=+0.01 (68.49)
20.97=0.255 (48.84)
19.55=0.16 (35.70)
2.284=0.26 (34.81)
Pad PreqLM
0.02=10002 (71.38)
20.98 (50.65)
19.71=0.04 (39.86)
23.56:10.60 (36.69)
Pad / IterLM
0.04 (72.98)
19.65=10.77 (45.76)
20.81=00066 (40.93)
22.79=1.20 (35.90)
Pad/ / /FllyLLM
0.13=00.02 (80.72)
25.37=0047 (53.71)
31.84=0.01 (67.72)
43.59=0007 (64.04)
Full RandLM
0.00++0.00 (70.13)
19.58+0081 (40.82)
4.00=:0.28 (49.60)
21.92:+4.01
(35.19)
Full / FreqLM
0.1010.08 (88.43)
24.36 1.77 (49.82)
20.87=10322 (66.64)
28.91=0045 (55.64)
Full / IterLM
0.12:10.08 (87.98)
19.93 (39.66)
23.40=1.72 (71.65)
2..6 (44.29)
Full FullyLM
(93.8))
04.39 (74.48)
38.52=00.01 (91.96) 5..09=0008 (89.62)
Table 2: The performance (mean (%) + std ( (%) comparison of VR with XCIT-S12 on CIFAR100, where PGD-10 used for adversarial training, and PGD-20 used for adversarial testing. The results are presented in the format of robustness (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model / VR using Adversarial Training
In / Out
Strategy (X x)
Strategy 2 (x / v)
Strategy 3 (V/ 1/// /x)
Strategy 4 (V/v)
P RandLM
0.33 (22.27)
1.10=0.14 (1.73)
1.13=00.9 (2.28)
0.79++0.01 (1.27)
Pad/ / FreqLM
0.10= (29.10)
0.54 (10.88)
3.06:+0.16 (6.35)
2.52++0.12 (4.05)
Pad/ / IterLM
0.141:001 (38.54)
0.75+10.02 (1.79)
4.61 (11.02)
5.26110.36 (8.90)
Pad/ FullFul
0.46++0.02 (58.32)
12.07=00.35 (36.35)
18.40++0.12 (44.29)
25.93++008 (43.59)
Full / RandLM
0.00 (19.15)
1.01 (1.15)
0.74 (6.32)
1.19:+0.20 (1.89)
Full / FreqLM
0.04 (49.84)
7.34+1990 (17.32)
9.66=0.13 (32.75)
00.62++.28 (30.49)
Full IterLM
0.18 (57.63)
2.74 (4.64)
11.29++0.17 (38.07)
11.93++.30 (33.21)
Full / FullyLM
0.78 (77.28)
24.55:10.20 (52.91)
22.02:+0.11 (74.06)
07.9 (72.88)
Table 2 and Table 3 present the performance comparison results of VR with pre-trained XCIT-S12 on CIFAR-10, CIF,R-100, and GTSRB, respectively.
Performance of VR with Naturally and Adversarially Pre-trained XCTT-S12. The results in- dicate that in Strategy 1, despite achieving very high clean accuracy, the robustness of various VR methods is nearly zero. For instance, when using naturally pre-trained XCTT-S12 in Strategy "Pad FullyLM" achieves a clean accuracy of 65.30% on GTSRB (Table 3h, yet its robustness is 0.57%; similarly, "Full IterLM" achieves clean accuracy of 57.63% on CIFAR-100 (Table 2 but also exhibits an adversarial robustness of 0.18%. Although in Strategy 1, "Full FullyLM' achieves clean accuracy of 93.84% and robustness of 12.09% on CIFAR-10 (Table [ the rest of the VR methods all achieve almost zero robustness. This demonstrates the extreme vulnerability of VR with naturally pre-trained models to adversarial attacks. Furthermore, when using adversarilly pre- trained XCIT-S12 in Strategy 3, although there is general decrease in clean accuracy compared with Strategy 1, most VR methods displayed some level of robustness. For example, on CIFAR- 10, "Pad/ / FullyLM' achieves clean accuracy of 67.72% with adversarial robustness of 31.84%; "Full IterLM" shows a clean accuracy of 71 71.55% with an adversarial robustness 23.40%. This suggests that existing VR methods can leverage the robustness of pre- pre-trained models to some extent.
Similarly, comparing the experimental results of Strategy 2 and Strategy 4, it is evident that the robustness of using adversarially pre-trained XCIT-S12 is mostly higher than using naturally
8

---

Published as a conference paper at ICLR 2025
pre-trained XCIT-S12. For instance, the robustness of "Pad RandLM" on CIFAR- 10 is 20.97% in Strategy 2. which is lower than 22.84% in Strategy 4. Consistent findings are also evident across other datasets and various VR methods.
These phenomenon is consistent with our theoretical analysis: the adversarial risk of VR in tar- get domain is upper-bounded by the adversarial margin risk of the pre-rained model in the source domain. Specifically, compared with naturally pre-trained models, adversarially pre-trained mod- els generally exhibit better robustness (see Appendix B), resulting in lower adversarial margin risk, which may help reduce the upper bound of adversarial risk in the target domain. Therefore, models reprogrammed using adversarially pre-trained models will have higher robustness compared whit those reprogrammed using naturally pre- re--trained models.
Performance of VR with Naturally and Adversarially Training. When using adversarial training in Strategy 2, there is also decrease in clean accuracy compared with Strategy (using natural training). Similarly, the robustness of VR in Strategy 2 show an improvement compared with Strat- egy 1. In fact, when we incorporate adversarial samples from target domain into the reprogramming process by adversarial training, we are directly optimizing the adversarial risk of VR, so the observed empirical phenomena are consistent with previous adversarial training research. For example, in the "Pad / FullyLM" on Table I the Strategy achieved a robustness improvement of 25.24% com- pared with Strategy 1; in the "Full/ / FullyLM', the Strategy 2 achieved robustness improvement of 32.30% compared with Strategy 1. Similar results can be observed in all experiments. This is consis- tent with the theoretical analysis that incorporating adversarial samples from the target domain task into the reprogramming process of VR through adversarial training may reduce the ADMM-VAL which can enhance the robustness of the reprogrammed models.
Similarly, by comparing the experimental results of Strategy and Strategy 4, it is evident that the robustness achieved through adversarial training during the reprogramming process higher than that achieved through natural training. It can be observed that the robustness improvements achieved through adversarial training during VR are relatively limited compared with those obtained using adversarially pre-trained models. For instance, as shown in Table the improvement of Strategy 4 over Strategy is generally smaller than that of Strategy over Strategy The limited robustness improvements from adversarial training may be due to the finite number of trainable parameters in VR. As illustrated Tables [ training becomes more pronounced when VR methods with larger parameter spaces (e.g.. SMM and Fu)lyLM) are employed. Therefore, adversarial training remains a crucial technique for improving robustness. Developing adversarial training methods tailored the VR framework and designing more effective fin and faur represent important challenges for enhancing the robustness of VR.
to 3 and Figure 3, the robustness enhancement through adversarial
Performance of VR under Different Adversarial Attack Intensities. We also evaluate the robust- ness of the VR under different adversarial attack intensities. Specifically, on the CIFAR- 10 and GT- SRB, we employ various VR methods using adversarially pre-trained ResNet-18 with TR TRADES as adversarial training. During the testing phase, we modify two key parameters of the PGD during the testing phase: setting the adversarial perturbation radius from (1/255, 2/255, 4/255, 6/ 6/255, 8/ 8/255] while fixing the iteration number at 20, and setting the number of iterations from {1. 10, 20, 50. 100} while fixing the adversarial perturbation radius at 4/255. The results are displayed in Figure As illustrated in Figure [3] although increasing either the adversarial perturbation radius or the number of PGD iterations leads to a reduction in the adversarial accuracy of the various VR methods, their relative robustness rankings largely remain unchanged. These results indicate that VR with an ad- versarially pre-trained model and using adversarial training remains robust when facing adversarial attacks of varying intensity.
Additionally, it can be clearly observed from Figure that, when four is fixed, the robustness gap caused by different fout is significantly smaller on CIFAR-10 compared with GTSRB This may be because both ImageNet and CIFAR-10 involve natural image classification, whereas GTSRB fo- cuses traffic signs, which differ from ImageNet in terms of style and context. Moreover, AMDD- VRA is influenced not only by the domain gap but also by the choice of fin and four- Consequently, when the domain gap is large, better fin can more effectively reduce these differences.
Interesting Phenomenon about Clean Accuracy. Moreover, we observed an interesting phe- nomenon: when performing adversarial training on a naturally pre-trained model by incorporating adversarial samples during reprogramming (i.c., Strategy 2). it achieves certain level of robustness

---

Published as a conference paper at ICLR 2025
Table 3: The performance (mean (%) std (%) comparison of VR with XCIT-SI2 on GTSRB, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustmess (clean accuracy), where the std of clean accuracy omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In / Out
Strategy (X /X)
Strategy 2 (x / v/)
Strategy 3 (V/ /X)
Strategy 4 (K/V)
Pad/ / RandLM
1.73 (44.11)
12.07=1001 (31.88)
5.82=0083 (10.22)
8.19=0.34 (12.96)
Pad FreqLM LM
0.24 (47.11)
14.22+00.56 (31.26)
8.43++0.10 (14.62)
9.49=1.40 (15.94)
Pad / IterLM
0.16=1.11 (47.54)
4.72=1.20 (9.15)
8.68=10227 (15.83)
9.79 (12.04)
Pad/ / FullyLM
0.57=0037 (65.30)
17.71 (44.09)
15.12=0.02 (43.36)
24.92=0.12 (42.76)
Full / RandLM
0.00 (74.41)
23.21 (44.66)
1.87 (40.64)
23.98= (37.37)
Full / FreqLM
0.00 (77.14)
11.60:11.01 (51.63)
5.70:10.11 (37.67)
27.04=1.71 (41.30)
Full IterLM
0.00 (76.03)
13. 13.006 (57.33)
5.15 (41.80)
29.58 +.58811.63 (48.34)
Full FullyLM
1.74 (88.70)
42.66-00.2 (87.89)
20.85 (80.62)
50.37=0.43 (79.88)
--- Pad / RerLM
Pad/ Fad/FullyLM
Full /IterLM
Full FullLM
SMM /IterLM
SMM / FullyUM
(35)
60-
Rebusttness (96)
Robustness (35)
-10
Robustness
30
92
10
10
Attack Radius
Attack Step
Attack Radius
Attack Step
Figure 3: The performance (robustness accuracy) of using TRADES for adversarial reprogramming an adversarially pre-trained ResNet- 18 model on the CIFAR- 10 and GTSRB datasets as the intensity of adversarial attacks increases.
but suffers a significant drop in clean accuracy. In contrast, when performing adversarial training on an adversarially pre-trained model by incorporating adversarial samples during reprogramming (i.e., Strategy 4), it maintains good level of robustness while also preserving certain clean accu- racy. For example, in Table 2 the clean accuracy of "Pad/ FullyLM' in Strategy 2 drops to 36.35% which lower than 43.59% of "Pad / FullyLM" in Strategy 4; the clean accuracy of "Full Ful- LyLM" in Strategy 4 is 72.88% higher than that of "Full / FullyLM" in Strategy 2. This phenomenon can observed in most of the experimental results, indicating that the effectiveness (in terms of robustness and clean accuracy) of adversarial training on naturally pre-trained models is inferior to that of adversarial training on adversarially pre-trained models.
6 CONCLUSION
In this paper, we for the first time investigate the adversarial robustness of VR, which is unexplored in previously visual reprogramming (VR) methods. Our empirical results reveal that employing nat- urally pre- -raitraind models and natural training are particularly vulnerable to adversarial attacks, while using pre-trained models with adversarial robustness and adversarial training can both significantly enhance the robustness of reprogrammed models. Additionally, we present a theoretically guaran- teed adversarial risk upper bound for VR, validating our empirical findings and laying a theoretical foundation for future research. Extensive experiments demonstrate the effectiveness of the strategies revealed in our empirical findings in enhancing the robustness of VR. Given the substantial potential of VR in diverse tasks and modalities, strategies hold potential applications in security-critical multimodal large language models. However, the robustness endowed by these strategies remains to be enhanced. In future research, will explore improved methods for VR based results.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTI
This work was supported by the National Key R&D Program of China under Grant J022YFB310300 Feng Liu is supported by the Australian Research Council (ARC) with grant number LP240100101, DE240101089 and DP230101540 and the NSF&CSIRO Responsible Al program with grant number 2303037.
Reproducibility Statement. For the datasets used in our paper, the detailed descriptions can be found in Appendix B For the experimental process in our paper, the complete implementation de- tails can be found in Appendix [ For the lemmas and theorems in our paper, the complete proofs can be found in Appendix A The code for the experiments can be found in supplementary material. These can help reproduce the theoretical and experimental results.
REFERENCES
Maksym Andriushchenko. Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: query-efficient black-box adversarial attack via random search, 2020.
Rob Ashmore, Radu Calinescu, and Colin Paterson. Assuring the machine learning lifecycle: Desiderata, methods, and challenges. ACM Computing Surveys (CSUR), 14(5): 2021.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models, 2022.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smdić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning at time. ECML PKDD, pp. 387-402, 2013.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative com- ponents with random forests. In David Fleet, Tomas Pajdla, Bernt Schicle, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014, pp. 446-461, Cham, 2014. Springer International Publish- ing.
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Sample-specific masks for visual reprogramming-bor prompting. In ICMI., pp. 1383-5508. 2024.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE Symposiun on Security and Privacy (SP), pp. 39-57, 2017. doi: 10. 10.1//9/P.2211.44.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural legal judgment prediction in english. In ACL, pp. 4317-4325, 2019.
Aochuan Chen, Peter Lorenz, Yuguang Yao, Pin-Yu Yu Chen, and Sijia Liu. Visual prompting for adversarial robustness, 2023a.
Aochuan Chen, Yuguang Yao, Pin-Yo Yu Chen, Yihua Zhang, and Sijia Liu. Understanding and im- proving visual prompting: A label-mapping perspective. In CVPR, pp. 19133-19143, 2023b.
Lingwei Chen, Yujie Fan, and Yanfang Ye. Adversarial reprogramming of pretrained neural net- works for fraud detection. In CIKM, pp. 2935-2939, 2021.
Pin-Yu Chen. Model reprogramming: Resource-efficient cross-domain machine learning. In AAAI, 2022.
Han Cheng, Wang Qifan, Cui Yiming, Cao Zhiwen, Wang Wenguan, Qi Siyuan, and Liu Dongfang. E2vpt: An effective and efficient approach for visual prompt tuning. In ICCV, 2023.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fiee free attacks, 2020.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateck Mittal, and Matthias Hein. Robustbench: a standardized adver- sarial robustness benchmark. In NeurlPS Datasets and Benchmarks' Track, 2021.

---

Published as a conference paper at ICLR 2025
Edoardo Debenedetti, Vikash Sehwag, and Prateek Mittal. A light recipe to train robust vision transformers. In SaTML, 2023.
Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross- covariance image transformers. arXiv, 2021.
Gamaleldin F. Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial reprogramming of neural networks, 2018.
Shuo Feng, Xintao Yan, Haowei Sun, Yiheng Feng, and Henry X Liu. Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment. Nature Communica- tions, 12(1):748, 2021.
Muhammad Ghifary, W. Bastiaan Kleijn, and Mengjie Zhang. Domain adaptive neural networks for object recognition. CoRR, abs/1409.6044 2014.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv, 2014.
Praveen Govi. CoronaHack-Chess X-Ray-Dataset, 2020.
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362-386, 2020.
Kanza Hamid, Amina Asif, Wajid Abbasi, Durre Sabih, al. Machine learning with abstention for automated liver disease diagnosis. In FIT, pp. 356-361, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, pp. 4700-4708, 2017.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Mohammad Ali Kadampur and Sulaiman Al Riyaee. Skin cancer detection: Applying deep learn- ing based model driven architecture in the cloud for classifying dermal cell images. Informatics in Medicine Unlocked. 18:002,,, 2020.
Eliska Kloberdanz, Jin Tian, and Wei Le. An improved (adversarial) reprogramming technique for neural networks. ICANN, pp. 3-15. Springer, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv, 2022.
Kangwook Lee, Changho Suh, and Kannan Ramchandran. Reprogramming gans via input noise design. In ECML PKDD, pp. 256-271, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Paarth Neekhara, Shehzeen Hussain, Jinglong Du, Shlomo Dubnov, Farinaz Koushanfar, and Julian McAuley. Cross-modal adversarial reprogramming. In WACV, pp. 2427-2435, 2022.
Maria-Elen Nilsback and Andrew Zisserman. Automated flower classification a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722-729, 2008
12

---

Published as a conference paper at ICLR 2025
Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199 210, 2011. doi: 10.
1109/TNN.20100209223.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, pp. 3498-3505, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, pp. 211-252, 2015. 2015.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenct models transfer better? In NeurlPS, pp. 3533-3545, 2020.
Naman D Singh, Francesco Croce, and Matthias Hein. Revisiting adversarial training for imagenet: Architectures, training and generalization across threat models. NeurlPS, 2023.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Bench- marking machine learning algorithms for traffic sign recognition. Neural Networks, 52:323-332 2012.
C Szegedy. Intriguing properties of neural networks. arXiv, 2013.
Yun-Yun Tsai, Pin- Yu Chen, and Tsung- Yi Ho. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In ICML, pp. 9614- 9624, 2020.
Hsi-Ai Tsao, Lei Hsiung, Pin-Yu Chen, Sijia Liu, and Tsung-Yr Yi Ho. Autovp: An automated visual prompting framework and benchmark. In ICIR, 2024.
Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of-distribution detection. In NeurIPS, pp. 15545-15557, 2022.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp- 3485-3492, 2010.
Chao-Han Huck Yang, Yun- Yun Tsai, and Pin- Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In ICML, pp. 1 11808-11 11819, 2021.
Zhewei Yao, Amir Gholami, Peng Xu, Kurt Keutzer, and Michael W. Mahoney, Trust region based adversarial attack on neural networks. In CVPR, June 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, pp. 7472-7482,
2019a.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In ICML., pp. 7404-7413, 2019b.
Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. Legal judgment prediction via topological learning. In EMNLP, pp. 3540-3549, 2018.
13

---

Published as a conference paper at ICLR 2025
A PROOFS OF THEADVERSARIAL RISK UPPER BOUND
A.1 PROOF OF LEMMA
Proof. We can discuss the following two scenarios based on whether the classification correct:
since the goal of adversarial attack is to induce incorrect predictions by the model, we are able to identify a 7 <I]>|| I <) such that ( + 7) + y] = and P1 (ær + 7, s 0. Therefore, we have | Se y (x: + 7) A v] = malle Φ pf)( + 7.3) 1.
1) If (x) &
2) If (x)) = if for any 7 <|]> - e) such that (x + 7) * y| = 0, then for any 2 0 we have max S. I 7) # yì = 0 < max]hils
pp) - 1; if there is a 7 (l17l S c) such
that [[/ + * y) the result is equal to scenario 1).
Combining 1) and 2), we have Rú R,dv.(P) ( 2 Ru (hy) for any p 2 0.
A.2 PROOF OF THEOREM
Lemma 2. For any distribution = p(x, x. y) and any sorce function f, we have:
dispe (dvv.
ap) (f') + R2 Redv. (e)))le)())) (f' ( S Rt
(13)
Proof. For any (a, y) U1 if there is a (//l S c) such that h + 7) o y or 1 +7) + y.
then we have me @ pf) + +7,8) = 1 or max ( pf)(x + 7,y) = 1, while
maxp/ (z) + 7, hy + 7)) $ 1. If for any (lyll I <) such that hp + 7) =
h + 7) = y, we have:
max (Ø) op +o+++00 +% (x
max ( pf: +7,9)
Isilss
max ( pp + 7.8) max (I, pf) 7,y). Sh
Ihlls
Combining the aforementioned two scenarios and taking expectations, we can obtain the result.
By combining Lemma we can proof Theorem 1:
Proof.
Rado (hfmmofsofma) Ey max I[h f foa (x 7) + yl Sh
(0)
5 ET max
(x 7) 4 fou (x + 7)] Io (x + 7) + [11}
Se
(6)
I By max Th (x 7) * h fon (x + 7) + max
Thoom (xr + 7) # y)
Whllse
1ylSe
ET max I
Iso (2 + 7) F hoot + 7) + ET max hhlss
Il (xr 7) *
Inlls
(0)
S disp
(ot (Fout o fS fins four o fs fin) + Rg R (fout f3 o fin)
disp oo (fout o f's Jin Sout is o fin)
adv.(p)
+ R (0)
(Sout o I3 o fin) + R
Radv.le) (Js) - R Radv.(o) (Js)
(d) adv.(e) (fout o f3 fins fout o fs o fin) + Redr (fout o fs fin)
2 disp
+ R (p) (Js) + R Radv, an)
(ʃī) - disps adv.
adv.(p)
(Sš. Is) (Lemma 2)
(e)
< Rs
(ddv.(0((fs) + Iso fin (S, T) + R 12adv
(d)
(fout o fs fun) +
14

---

Published as a conference paper at ICLR 2025
where f3 arg min [ {R' Ri...oo (Js)) and X Rs R
(S$) is constant associated with the source
fsEFs domain. The proof is completed.
B ADDITIONAL INFORMATION INOORMOTAAIFFORAATIOOON OF EXPERIMENTS
B. ADVERSARIAL EXAMPLE GENERATION
In adversarial training (e.g., PGD-10 TRADES) and validating model robustness, employ the iterative projected gradient descent method to generate adversarial examples. Assuming the total number of iterations N, the adversarial example step S N is represented as follows:
adv = II < + sign V ae LNT (f( 10),)).
(14)
where the II denotes the porjection, a denotes the step size, and sign(-) evaluates to -1, 0, or +1 for each component of the gradient, depending whether the component is negative, zero, positive, respectively. Assuming that æ represents an image, the projection of the value in i-th row and j-th column of c-th channel is follows:
a), adv [ J. c (alpaaraal (. a],, c.c e, aiii + e),
(15)
where denotes the perturbation radius and clip denotes the operation that restricts the value within a specified range. Similarly, TRADES, adversarial examples are obtained by maximizing the KL divergence follows:
x f++ adv
11 (x + a sign( Vas, (KL (f(o 10) 0)))
(16)
The detailed process is presented in Algorithm
Algorithm Adversarial Example Generation using Iterative Projected Gradient Descent
Input: Original input ær, True label y. Model parameters 0, Loss function (NT, Step size a, Pertur-
bation radius e, Total iterations N
Output: Adversarial example æadv
1: Initialize 9 + x
2: for t = to N do
3: Compute gradient: of ndv 10),
4: Update adversarial example:
5: 1+1 +- x + a sign (aay, eNT (x ()).))
Fddy
6: Project onto e-ball around x:
7:
for each pixel (i, c) do
8:
x [ū, j, +- clip (o ( j.cl. c), æli,j.c æli.j.d +e)
9: end for
10: Fady adv
11:
end for
Return Kndv
Table 4: Performance of different pre- traitrained models source domain (ImageNet- Robustness represents the accuracy of PGD-20 with = 4/255 and = 8/255 under Tese-oorm.
Adversarially Pre-trained Model
Performance
RN18 (X)
RN50 (X)
XCIT-S12 (X)
RN18 (V)
RN50 (V)
XCIT-S12 (V)
Clean Acc.
69.75
76.13
82.00
48.42
60.49
70.60
Robustness = 4/255)
0.00
0.00
0.00
26.07
35.45
39.86
Robustness (e 8/255)
0.00
9.88
14.26
14.30
15

---

Published as a conference paper at ICLR 2025
Table 5: The performance (mean (%) + std (%)) comparison of VR with ResNet-18 on CIFARIO, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness (clean accuracy), where the std of clean accuracy omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In / Out
Strategy (X /X)
Strategy 2 (X //)
Strategy 3 (V/ / x)
Strategy 4 (V/ /V)
Pad/ / RandLM
0.12=0.02 (56.72)
8.24:0.71 (25.86)
17.37=1.17 (26.59)
17.66=1.27 (24.98)
Pad FreqLM LM
0.12400.04 (58.41)
6.62++.66 (26.22)
18.69+0.83 (29.69)
20.92++0551 (29.18)
Pad / IterLM
0.14 (58.69)
4.70++2.43 (26.62)
20.01 (32.49)
20585 (30.83)
Pad/ / FullyLM
0.11 (71.62)
17.59+0.25 (40.16)
32.15+0.16 (56.70)
37.72=0.12 (53.61)
Full M
0.00 (61.78)
3.04 (29.68)
1.51 (41.13)
11.38 (28.16)
Full / FreqLM
0.00 (70.15)
3,58 (32.25)
13.8510.40 (56.05)
18.64=0.07 (45.69)
Full IterLM
0.00 (69.22)
4.45 (30.25)
17.77-14.22 (58.01)
13.78 (39.91)
Full FullyLM
0.40=#0.03 (86.18)
24. (44.10)
35.20 (87.15)
0..1 (82.92)
Table 6: The performance (mean (%) std (%)) comparison of VR with ResNet-18 on CIFAR100,
where PGD-10 used for adversarial training, and PGD-20 used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy omitted. The best robustness highlighted in bold.
Adversarially Pre-trained Model / VR using Adversarial Training
In Out
Strategy (X /X)
Strategy 2 (X / V)
Strategy 3 (V/) / /X)
Strategy 4 (V/V)
Pad/ RandLM
0.06+00.04 (8.42)
0.46++0.0 (1.05)
0.79::0.24 (1.42)
0.78++0.33 (1.16)
Pad / FreqLM
0.06=1001 (13.46)
0.28110221 (1.35)
2.11 (3.69)
1.92-10.15 (2.74)
Pad / IterLM
0.09=0002 (20.41)
0.64 (1.45)
5.42:0028 (8.96)
5.04:+0.27 (7.58)
Pad / FullyI.M
0.29++0.04 (46.81)
6.36++0.09 (23.87)
18.37 (34.24)
21.17:+0.2 (32.61)
Full RandLM
0.00=0.00 (12.19)
0.18::0.29 (0.89)
0.82=0.07 (4.40)
0.64+0.22 (1.27)
Full / FreqL.M
0.00++000 (29.79)
0.07++0.33 (2.06)
=.91 (25.50)
8.50+0.04 (23.42)
Full IterLM
0.00+10.00 (36.87)
0.50+0.15 (3.54)
10.25 (31.77)
10.88:1012 (28.84)
Full / FullyLM
0.07 (68.33)
11.87+00.7 (30.31)
21.91 (67.75)
36.57+0.12 (65.55)
B.2 EVALUATION METRICS
In the testing phase, we employ the classification accuracy on adversarial test examples as the ro- bustness, where the adversarial test examples are generated from clean examples in the test dataset using 20-step Projected Gradient Descent (PGD-20) with a perturbation radius of e 4/255 un- der the -nom. Let the classification accuracy of the clean test examples be the clean accuracy. In addition, we provide the results for different perturbation radii and different iteration steps in Figure
B.3 IMAGENET-IK
This is the most commonly used subset of the famous image classification dataset ImageNet (Rus- Sakovsky et al. 2013). The dataset covers 1,000 object classes and includes 1,281,167 training images, 50,000 validation images, and 100,000 test images. We set this task as our source domain because the dataset has large amount of training examples and wide variety of classification cat- egories, which enables the pre-training of excellent models. Based on this, we use three preitrained models on this dataset for reprogramming: ResNet-18 (RN18), ResNet-50 (RN50) (He et al. 2016) and XCTT-S12 (El-Nouby et al. 202T). Their input layers are all 224 x 224. The weights of the naturally pre-trained models (ResNet-18 (N.T.) and ResNet-50 (N.T.)) are obtained from the official
16

---

Published as a conference paper at ICLR 2025
Table 7: The performance (mean (%) std (%) comparison of reprogrammed ResNet-18 on GT- SRB, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In Oul
Strategy (x X)
Strategy 2 (X / V)
Strategy 3 (V/) X))
Strategy 4 (V/v))
Pad/ / RandLM
0.94 (30.49)
0.82:10.62 (11.19)
5.12::0.20 (7.01)
4.09+1.88 (5.87)
Pad FreqL.M
0.22++0.3 (35.02)
1.96+0.24 (15.07)
6.23++0.33 (10.48)
5.91+0.20 (9.04)
Pad/ / IterLM
0.21 (37.70)
4.95+0031 (6.15)
7.17=0.40 (10.30)
8.44 (11.33)
Pad FullyLM
1.47::0266 (58.84)
11.37:10.30 (37.80)
12.70:10.04 (29.01)
17.27:1.11 (28.55)
Full M
0.00++0.00 (66.37)
0.31 (8.13)
2.24+1128 (25.06)
4.54+2283 (14.57)
Full / FreqLM
0.00=00000 (68.90)
3.36:12.79 (11.45)
7.79=0096 (31.50)
8.77=00.4 (23.66)
Full IterL.M
0.00=0000 (70.37)
2.22++1.63 (21.65)
1.90= (35.46)
10.69+1.34 (27.98)
Full FullyLM
8.48+0.13 (89.31) 37.16 (76.92)
16.32:+0.08 (78.33)
0924 (69.00)
Table 8: The performance (mean (%) + std (%) comparison of VR with XCTT-S12 CIFARIO, and PGD-20 is used for adversarial testing. Using TRADES (A = 6) as adversarial training. The results presented in the format of robustness (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In / Out
Strategy 1 (X / X)
Strategy 2 (X
Strategy 3 (V/ / x)
Strategy 4 (V/ /v)
Pad/ / IterLM
0.04 (72.98)
15.44.10.38 (58.83)
20.81=10066 (40.93)
25.67=10.44 (41.78)
Pad / FullylL
0.13:1002 (80.72)
19.82:10.31 (63.51)
31.84=0.01 (67.72)
43.04=10.88 (65.01)
Full IterLM
0.12:40.08 (87.98)
29.51 (57.09)
23.40=0072 (71.65)
30.08=1.266 (52.04)
Full FFullyLM
12.09=0.74 (93.84)
48.02++0.12 (73.84) 38.52-+001 (91.96)
56.31 (82.66)
Table The performance (mean (%) std (%) comparison VR with XCTT-S12 on CIFAR100,
and PGD-20 is used for adversarial testing. Using TRADES (A = 6) as adversarial training. The
results are presented the format of robustness (clean accuracy), where the std clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In / Out
Strategy (X /X)
Strategy (X V)
Strategy 3 (V/ /X)
Strategy 4 (V /V)
Pad / IterLM
0.1411001 (38.54)
3.13::022 (24.71)
4.611:0.4 (11.02)
5.74:10.14 (10.99)
Pad/ / FullyLM
0.46=1002 (58.32)
11.35 (46.62)
18.4040.12 (44.29)
25.77=10.33 (44.24)
Full / IterLM
0.18:1.03 (57.63)
8.42+43.58 (26.05)
11.29:10.17 (38.07)
6.37-0.21 (16.47)
Full FFullyLM
0.78=0.13 (77.28)
27.24 (52.63)
22.02+0.11 (74.06)
38.47=0.18 (64.45)
PyTorch model repository, while the weights of the adversarially pre-trained models (ResNet- 18 (A.T.), ResNet-50 (A.T.) (Salman et al.] (202)) and XCTT-S12 (Debenedetti et al. 2023)) are ob-
tained from Robustbench (Croce et al. 2021). The performance of these pre-trained models on the source domain can be found in Table
B.4 CIFAR100
We conduct experiments using the well-known image classification dataset CIFAR- 100 (Krizhevsky al. 2009) target domain task, which consists 100 categories. The dataset consists of 60,000
17

---

Published as a conference paper at ICLR 2025
Table 10: The performance (mean (%) = std (%)) comparison of VR with XCIT-SI2 on GTSRB,
and PGD-20 is used for adversarial testing. Using TRADES (A 6) as adversarial training. The results are presented in the format of robustness (clean accuracy), where the std of clean accuracy omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In / Out
Strategy (X /X)
Strategy 2 (x / v/)
Strategy 3 (V/ XX)
Strategy 4 (V)
Pad / IterLM
0.16 (47.54)
11.36:+105 (36.75)
8.68=0.27 (15.83)
8.84=0078 (14.18)
Pad/ / FullyI.M
0.63++0.77 (65.30)
15.74:+0.05 (49.85)
12 (43.36)
2.014 (43.18)
Full IterLM
0.00=10.00 (76.03)
41.14:10.22 (68.44)
5.15::0.45 (4180)
28.70=10.02 (49.65)
Full / /FullyL
1.74+0.21 (88.70)
53.66++0.05 (84.35)
20.85=0.25 (80.62)
0.32 (76.97)
Table 11: The performance (mean (%) = std (%)) comparison of VR with ResNet-18 on CIFARIO. and PGD-20 is used for adversarial testing. Using TRADES (A 6) as adversarial training. The results are presented in the format of robustness (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model / VR using Adversarial Training
In Out
Strategy (X x))
Strategy 2 (X v/)
Strategy 3 (V/ / ())
Strategy 4 (V/v)
Pad/ / IterLM
0.14 (58.69)
2.93:+0.23 (39.56)
20.01 (32.49)
20.60 (32.22)
Pad / /FllyLMM
0.11 (71.62)
9.50++0.20 (54.76)
32.1550.116 (56.70)
37.18=0014 (54.01)
Full IterLM
0.00 (69.22)
0.09: (25.40)
17.77+14.22 (58.01)
10.41 (34.61)
Full Full/Full FullyLM
0.40=10.03 (86.18) 11.54:10.98 (55.93)
35.20:1.:21 (87.15)
53.68:10299 (77.94)
Table 12: The performance (mean (%) - std (%)) comparison of VR with ResNet-18 on CIFAR100,
and PGD-20 is used for adversarial testing. Using TRADES (A 6) adversarial training. The results are presented in the format of robustness (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
Out /
Strategy (X XX)
Strategy (X //)
Strategy (V/) /X)
Strategy (V/ /V)
Pad/ ItrLM
0.09 (20.41)
0.44 (9.18)
5.42: (8.96)
5.26:10.34 (8.15)
Pad/ / FullyLM
0.29:4000 (46.81)
3.49:+0.15 (36.76)
18.37:0.19 (34.24)
21.02=0.12 (33.15)
Full IeeLLM
0.00+0.00 (36.87)
0.01 (3.64)
10.25: (31.77)
8.20+#0 14 (22.83)
Full FullyLM
0.07 (68.33)
7,00::0.16 (39.36)
21.91 (67.75)
36.88=0004 (61.32)
32 x 32 pixel color images (50,000 images for training and 10000 images for testing). We followed the same setup. taking the original training data the training data and using the method in Ap- pendix to generate the adversarial training data. The optimizer is Adam W. The learning rate of the optimizer is searched from set {1 10 5 x 10 10- while the weight decay of the optimizer is searched from set {10-2 2 10--1----- $3. Moreover, we use this optimizer to train pre-trained model for 60 epochs, with the learning rate decreased 30 epochs and 50 epochs, respectively. Furthermore, in T we report the results of reprogramming pre-trianed ResNet- 18, 88 where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. In Table [2 we report the results of reprogramming a pre-trianed ResNet-18. 18, where TRADES (A = 6) is used for adversarial training, and PGD-20 is used for adversarial testing. Table [5 we report the results of reprogramming a pre-trianed ResNet- 18, where PGD-10 under 8/255 is used for adversarial training, and PGD- 20 under = 8/255 is used for adversarial testing. These results also demonstrate the effectiveness of our strategy for enhancing the robustness of the reprogrammed model.
18

---

Published as a conference paper at ICLR 2025
Table 13: The performance (mean (%) = std (%)) comparison of VR with ResNet-18 on GTSRB, and PGD-20 is used for adversarial testing. Using TRADES (A 6) adversarial training. The results are presented in the format of robustness (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model / VR using Adversarial Training
In Oul Out
Strategy 1 (x X)
Strategy 2 (X / v)
Strategy 3 (V/) /X)
Strategy 4 (V/V/)
Pad / IterLM
0.31 (37.70)
3.17::0.85 (13.50)
7.17=0.40 (10.30)
7.18:10.77 (9.94)
Pad FullyLM
1.47:=0.26 (58.84)
6.99++0.27 (38.42)
12.70= (29.01)
16.87:+0.09 (27.76)
Full IterLM
0.00=00.00 (70.37)
0.00=+0.00 (11.98)
7.90+ (35.46)
7.67 (26.89)
Full / FullyLM
8.48 (89.31)
25.36+1.41 (64.62)
16.32:10.08 (78.33)
41.45::0.01 (64.62)
Table 14: The performance (mean (%) = std (%)) comparison of VR with ResNet- 18 on CIFARIO, where PGD-10 (e = 8/ 8/255) is used for adversarial training, and PGD-20 (e = 8/255) is used for adversarial testing. The results are presented the format of robustness (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In Out
Strategy (X / x)
Strategy 2 (X /
Strategy 3 (v/)) IV/X
Strategy (V/ /V)
Pad / RandLM
0.00+0.00 (56.72)
3.9611131 (22.05)
10.51 (26.59)
13.61:0.86 (24.96)
Pad/ FrogLM
0.00:+0.01 (58.41)
2.33 (22.52)
10.6811.69 (29.69)
14.97:1.49 (27.20)
Pad IterLM
0.0010.00 (58.69)
0.50 (22.30)
10.7310.69 (32.49)
16.53:1.54 (30.69)
Pad / FullyLM
0.00++0.00 (71.62)
15.86:20.32 (20.49)
13.62:0.11 (56.70)
28.39-10066 (49.52)
Full / RandLM
0.0010.00 (61.78)
1.36:1022 (25.23)
0.22=0.14 (41.13)
8.02.1335 (21.44)
Full / FreqLM
0.00+1.00 (70.15)
1.57+1135 (23.44)
1.39-0.11 (56.05)
6.44 (29.92)
Full IteLLM
0.00+0.00 (69.22)
1.56+ (24.23)
2.42-0.86 (58.01)
7.62+ (30.64)
Full / FullyLLM
0.25+0.02 18)
18.35+00.23 (32.54)
5.37 (87.15)
31.34+0.18 (73.43)
Table 15: The performance (mean (%) std (%) comparison of VR with ResNet-18 on CIFAR where PGD-10 = 8/ 8/255) used for adversarial training, and PGD-20 = 8/255) is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness highlighted in bold.
Adversarially Pre-trained Model / VR using Adversarial Training
In Out
Strategy 1 (X / x))
Strategy 2 (X ()
Strategy 3 (V/ /
Strategy 4 (V/v)
Pad/ / RandLM
0.00+10.00 (8.42)
0.01+0002 (1,20)
0.41-10.11 (1.42)
0.77:10.09 (1,29)
Pad/ / FreqLM
0.00 (13.46)
0.37+0.54 (1.02)
1.21=0.19 (3.69)
1.07+t.33 (2.08)
Pad/ IterLM
0.01+10.01 (20.41)
0.51 (1.33)
3.28 (8.96)
3.7410.28 (5.79)
Pad Fullyu
0.00 (46.81)
2.98+0.07 (8.40)
8.75=0.03 (34.24)
14.77+0.13 (30.58)
Full / RandLM
0.00 (12.19)
0.2710.41 (1.02)
0.09 (4.40)
0.51:10.36 (1.26)
Full / FreqL.M
0.00++0.00 (29.79)
0.00+ (1.37)
1.76=0.06 (25.50)
1.83+00113 (18.36)
Full IterLM
0.00 (36.87)
0.15#0.14 (2.78)
2.18 (31.77)
1.89+0.21 (19.85)
Full / FullyLM
0.06++0.01 (68.33)
6.62++0.99 (15.51)
0.05 (67.75)
20.30++.12 (57.80)
B.5 CIFARIO
CIFAR-10 (Krizhevsky et al., 2009) is also a renowned image classification dataset, similar to CIFAR-10W 100. Its data configuration nearly the same that of CIFAR-100 (consisting of 60,000 32
19

---

Published as a conference paper at ICLR 2025
Table 16: The performance (mean (%) + std (%)) comparison of reprogrammed ResNet-18 on GT- SRB, where PGD-10 (c = 8/255) is used for adversarial training, and PGD-20 (c 8/255) is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accu- racy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In / Out
Strategy 1 (X (X)
Strategy (x v)
Strategy (V/) /
Strategy 4 (V/V)
Pad/ / RandLM
0.03+00044 (30.49)
0.04+:000 (9.20)
3.76=+0.33 (7.01)
4.01=1.07 (7.21)
Pad/ / FregLM LM
0.00+0000 (35.02)
0.20++0.1 (10.35)
3.68+0.67 (10.48)
4.29- (9.13)
Pad / IterLM
0.00 (37.70)
3.11+2.75 (6.66)
5.02=0.36 (10.30)
6.74-0.51 (9.74)
Pad/ / /FllyLLM
1.05=0118 (58.84)
6.05++028 (14.84)
5.40=0.06 (29.01)
12.04=0.09 (25.58)
Full / RandLM
0.00+0000 (66.37)
0.01 (6.38)
0.13:0.12 (25.06)
1.70= (10.06)
Full / FreqLM
0.00 (68.90)
0.68:10.30 (6.10)
1.26= (31.50)
3.24 (16.18)
Full / IterLM
0.0010.00 (70.37)
1.1540067 (8.99)
1.77=0.32 (35.46)
0.99 (20.54)
Full FullyLM
6.66+0.44 (89.31)
24.81 (60.31)
6.26=0017 (78.33)
0.09 (59.07)
Table 17: The performance (mean (%) = std (%)) comparison of VR with ResNet- 18 on CIFARIO, and PGD-20 = 8/255) is used for adversarial testing. Using TRADES (A 6 and 8/255) as adversarial training. The results are presented in the format of robustness (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In Out
Strategy (X XX)
Strategy (X /
Strategy 3 (V/ /X)
Strategy (V/ (V)))
Pad/ / IeeLL
0.00+ (58.69)
0.61 (36.79)
10.73++0.69 (32.49)
14.42-+0.9 (33.30)
Pad / FullyLM
0.00+0.00 (71.62)
2.28=0.20 (50.26)
13.62++0.1 (56.70)
25.44=10.06 (50.62)
Full IterLM
0.00+0.00 (69.22)
0.02=0.04 (24.69)
0.42 (58.01)
0.33=0067 (30.02)
Full /FullLLM
0.25+ 18)
0.87=0.80 (46.68)
5.37 (87.15)
30.43:10.04 (71.11)
x 32 pixel color images, with 50,000 images for training and 10,000 images for testing), except that CIFAR-10 has only 10 classification categories. We use the same optimizer as for CIFAR-100 for reprogramming. In Table we report the results of reprogramming a pre-trianed ResNet-18, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. In Table [1] we report the results of reprogramming pre-trianed ResNet- where TRADES (A 6) is used for adversarial training, and PGD-20 is used for adversarial testing. In Table [4 we report the results of reprogramming pre-trianed ResNet-18, where PGD-10 under = 8/255 is used for adversarial training. and PGD-20 under = 8/255 is used for adversarial testing. These results also indicate that incorporating adversarial samples from the target domain and using adversarially pre-trained trained models can both enhance the robustness of the reprogrammed model. In addition, we present some visual results of the reprogramming Figures and
B.6 GTSRB
The German Traffic Sign Recognition Benchmark (GTSRB) (Stalkkapp et al.) 2012) contains 43 classes of traffic signs, split into 39,209 training images and 12,630 test images. Due to the smaller image size and fewer classification categories compared with IMAGENET-IK, we use this dataset as our target domain task for reprogramming. The images have varying light conditions and rich backgrounds. We continue to use their data partitioning for training and testing. We use the same optimizer as for CIFAR-100 for reprogramming. In Table 7 we report the results of reprogramming a pre-trianed ResNet- 18, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. In Table 13, report the results of reprogramming a pre-trianed ResNet- 18, where TRADES (A = 6) is used for adversarial training, and PGD-20 is used for adversarial testing. In Table 14
report the results of reprogramming a pre-trianed ResNet-18 18, where PGD-10 under
20

---

Published as a conference paper at ICLR 2025
Table 18: The performance (mean (%) std (%) comparison of VR with ResNet-18 on CIFARI00, and PGD-20 (c 8/ /555)) is used for adversarial testing. Using TRADES (A = 6 and c 8/255) as adversarial training. The results are presented in the format of robustmess (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In Out
Strategy (x /
Strategy (X (
Strategy 3 (V/) /X)
Strategy 4 (V/v)
Pad/ / IterLM
0.01 (20.41)
0.05 (7.95)
3.28 (8.96)
3.66:10.09 (8.09)
Pad / FullyLM
0.00 (46.81)
0.49+00.9 (34.64)
8.75=0003 (34.24)
13.36:+.04 (31.97)
Full IterLM
0.00 (36.87)
0.00+ (3.38)
2.18 (31.77)
0.55400.0 (4.57)
Full FullyLM
0.06=10.01 (68.33) 2.17=00.3 (31.97) 0.05=0.05 (67.75)
19.02:0.07 (56.02)
Table 19: The performance (mean (%) = std (%)) comparison of VR with ResNet-18 on GTSRB, and PGD-20 (e = 8/ 2255) is used for adversarial testing. Using TRADES (A = 6 and e 8/255) as adversarial training. The results are presented in the format of robustness (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In Out
Strategy 1 (X x)
Strategy (X v)
Strategy (V/) /
Strategy 4 (V/v)
Pad / IterLM
0.00+0000 (37.70)
0.47+0006 (11.64)
5.02=0036 (10.30)
6.02=0091 (10.52)
Pad/ / FullyLM
1.05+0.18 (58.84)
0.93++0.07 (34.88)
5.40=0.06 (29.01)
10.46::0.0 (25.70)
Full IterLM
0.0010.00 (70.37)
3.80:13.29 (6,33)
1.77:0.32 (35.46)
2.03 (17.37)
Full / FullyLL
6.6610.44 (89.31)
12.85400.4 (57.77)
6.2611017 (78.33)
2..74 (55.32)
8/255 is used for adversarial training, and PGD-20 under 8/255 used for adversarial testing. These results also validate our strategy.
B.7 ADVERSARIAL TRAINING WITH LARGER PERTURBATION RADIUS
We keep other settings unchanged and set the perturbation radius for adversarial training and testing e 8/255 further validate our analytical results. Tables 14 to 19 present the results of ResNet- 18 on different datasets. We observe that these results exhibit slighlly lower robustness, which may be attributed to the relatively high attack intensity at = 8/255. Furthermore, as shown in Table 4 although the adversarially pre-trained model demonstrates higher robustness under = 4/255, 8/255. This leads excessively high adversarial risk for the pre-trained model in the source domain. According to our theoreti- cal analysis, this elevated risk may result increased adversarial risk in the target domain after reprogramming. Nevertheless, these results align well with our analysis.
its robustness significantly degrades when facing attacks
C EXPERIMENTS ON COMPLEX MODELS AND LARGE DATASETS
To further validate the effectiveness of utilizing adversarially pre-trained models and employing ad- versarial training during the reprogramming process, we conducte experiments using adversarially pre-trained models on ImageNet, including XCTT-S12 and ViT-S+ConvStem (Singh et al. 2023).
which feature more complex architectures compared with ResNet-18 and ResNet-50. Additionally,
we perform experiments on larger datasets, including Flowers 102 NNilsback Zisserman 2008
OxfordPets (Parkhi al. 2012). SUN397 (Xiao et al. 2010), and Food101 (Bossard et al. 2014).
which offer higher image resolutions compared with CIFAR- 10, CIFAR- 100, and GTSRB More- over, SUN397 and F00 provide more extensive training and validation data. Following the setup in (Chen et al. 20235), we resize the images to a resolution of 128 x 128, while keeping all other experimental settings consistent with those described above.
21

---

Published as a conference paper at ICLR 2025
Table 20: The performance (mean (%) std (%)) comparison of reprogrammed XCIT-S12 on Flow- ers ers102, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In/ Out
Strategy 1 (x/ /
Strategy 2 (x / v)
Strategy 3 (V/) /X)
Strategy 4 (V/v)
Pad/ / IterLM
0.00 (6.64)
0.00= (6.40)
1.83::.24 (6.53)
1.86:1:.22 (6.57)
Pad/ / FullyLM
0.05++0007 (66.60)
0.00==0.00 (21.26)
29.06+#0.88 (64.47)
37.16:+0.48 (66.25)
Full IterLM
0.00=10.00 (6.90)
0.00 (5.85)
0.8610.07 (6.20)
1.53:10.30 (5.83)
Full / FullyLM
0.00#+0.00 (65.55)) 0.04=:0.06 (26.85)
26.46::0.94 (70.17) 39.93+40.0 (72.35)
The performance (mean (%) std (%) comparison of reprogrammed XCTT-S12 on Ox- fordPets, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted bold.
Table 21:
Adversarially Pre-trained Model VR using Adversarial Training
In Out
Strategy 1 (x / x)
Strategy 2 (X / v))
Strategy 3 (V/) M//x/ XX)
Strategy 4 (V/v)
Pad/ / IterLM
0.00+0.00 (70.94)
0.04:10.02 (38.90)
30.82:+061 (61.23)
30.85:+0.57 (59.74)
Pad / FullyLM
0.0110002 (86.37)
0.361:051 (5.19)
28.32:10008 (77.37)
37.23:10.02 (78.00)
Full IterLM
0.00++0.00 (72.01)
0. (37.33)
28.61 (66.87)
41.02-+0.36 (67.37)
Full / FullyLM
0.00 (87.63)
4.00:10.06 (35.97)
(85.
47.14:10.18 (86.55)
Table 22:
The performance (mean (%) + std (%)) comparison of reprogrammed XCIT-S12 on SUN397, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In Out
Strategy (X X)
Strategy (X V)
Strategy 3 (V/ /X)
Strategy 4 (V/V)
Pad/ IterLM
0.00 (30.49)
2.194:0:3 (2.19)
5.6311005 (17.27)
5.42=0.14 (10.85)
Pad / FullyLM
0.18 (59.19)
2.38 (2.80)
8.68+00.7 (49.90)
22.32:+0.08 (51.07)
Full / IterLM
0.00+10000 (25.97)
2.1940.03 (2.19)
4.8810.10 (20.99)
5.72:10.31 (16.27)
Full / FullyLM
0.23 (59.31)
6.71 (20.02)
7.97 (56.30)
24.31:10112 (58.49)
C.I. DESCRIPTION OF DATASETS
Flower102 comprises 102 flower species commonly found in the United Kingdom, with each cat- egory containing between 40 and 258 images. The dataset exhibits significant variations in scale, pose, and lighting conditions, as well as intra-class diversity and inter-class similarity. It is divided into training, validation, and test sets, with the training and validation sets each containing ages per class (1,020 images in total), while the test set includes the remaining 6, 6,149 images (at least 20 images per class), OxfordPets consists of 37 pet categories, each represented by approx- imately 200 images, and is split into 3,680 training and 3,669 test images. SUN397, designed for scene understanding research, contains 108,753 images spanning 397 categories, with each category including minimum of 100 images. A random 75% of the data is allocated for training, with the remaining 25% reserved for testing. Finally, Food101 includes 101 food categories with a total of 101, 100000 images. Each category comprises 750 uncleaned training images and 250 manually curated
im-
22

---

Published as a conference paper at ICLR 2025
The performance (mean (%) std (%)) comparison of reprogrammed XCIT-S12 on Food101, where PGD-10 is used for adversarial training. and PGD-20 is used for adversarial test- ing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Table 23:
Adversarially Pre-trained Model VR using Adversarial Training
In Out
Strategy 1 (x /
Strategy 2 (X v)
Strategy 3 (V /x)
Strategy 4 (V/v)
Pad/ / IterLM
0.00 (24.56)
0.24:10.31 (1.00)
2.6110.11 (11.45)
1.56=1.33 (4.96)
Pad/ FullyLM
0.03 (67.00)
0.83:10.07 (1.00)
9.7911003 (50.35)
21.58:10.03 (45.72)
Full / IterLM
0.00 (19.32)
0.93:10.05 (1.02)
0.74100.0 (10.26)
0.61=0.51 (1.07)
Full / FullyLM
0.05+1000 (64.69)
5.82:1.33 (12.30)
9.27:t0.03 (57.77)
23.26::0044 (52.80)
Table 24: The performance (mean (%) + std (%)) comparison of reprogrammed ViT-S+ConvStem on Flowers102, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model / VR using Adversarial Training
In Out
Strategy 1 (x X)
Strategy 2 (X v/)
Strategy 3 (V/
Strategy 4 (V/v)
Pad IterLM
0.00 (8.72)
0.00 (3.12)
2.31 (7.01)
2.34:10.05 (7.06)
Pad / FullyLM
0.02:10.02 (55.20)
0.76:10.0 (33.06)
33.72::0.18 (63.66)
39.67:1.31 (64.44)
Full IterLM
0.0010.00 (7.12)
0.00+1.00 (4.36)
2.04:10.15 (8.81)
1.77=10.07 (7.15)
Full Hall/uul FullyLM
0.00+1.00 (68.47)
3.11 (50.68)
34.91 (68.42)
44.08:10.9 (71.69)
Table 25: The performance (mean (%) + std (%) comparison of reprogrammed ViT-S+ConvSSem on OxfordPets, where PGD-10 used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model / VR using Adversarial Training
In Out
Strategy 1 (X / x)
Strategy 2 (X / V)
Strategy 3 (V/ / /X)
Strategy 4 (V/v)
Pad IterLM
0.00++000 (60.73)
0.00++0.00 (29.17)
34.99:-0.16 (59.24)
35.59:+065 (59.58)
Pad / FullyLM
0.00 (75.13)
0.21+0002 (39.65)
35.82+0.31 (77.10)
39.88:1024 (76.47)
Full IterLMM
0.00=0000 (70.33)
0.15+0002 (3.66)
34.84=10.45 (64.69)
44.00++0112 (65.56)
Full / FullyLM
0.00+10.00 (85.48) 11.29410.0 (53.93) 43.23 (82.82)
49.33:10.12 (83.04)
test images. The training images contain degree of noise, including intense colors and occasional mislabeling, while all images are rescaled to a maximum side length of 512 pixels.
C.2 EXPERIMENTAL RESULTS
We present these experimental results in Tables 20 27 which demonstrate the effectiveness of our strategy on more complex models as well as on larger datasets.
C.3 EXPERIMENTS ON MEDICAL DATASETS
We also conduct experimental on the ChestX-Rar Govi 2020) benchmark dataset (5,933 2D ra- diographs: 5,309 training/624 testing) containing healthy subjects and six etiological pulmonary infection subtypes, demonstrating our strategy's effectiveness in safety-criticn medical imaging di- agnostics. The results in Table 28 remain consistent with main conclusions.
23

---

Published as a conference paper at ICLR 2025
Table 26: The performance (mean (%) std (%)) comparison of reprogrammed ViT-S+ConvStem on SUN397, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In Oul
Strategy (x / X)
Strategy 2 (x / v)
Strategy 3 (V/) /X)
Strategy 4 (V/v)
Pad/ / IterLM
0.02+10.00 (23.41)
1.3 (2.06)
8.56=0.16 (20.59)
9.91:10.29 (20.20)
Pad FullyLM
0.00+0001 (52.16)
4.04-+0.13 (15.42)
11.92:#0.13 (51.51)
24.17:10112 (53.50)
Full IterLM
0.00=00.00 (27.29)
2.19+ (2.19)
6.71= (21.82)
8.63++0.01 (19.06)
Full / FullyLM
0.01 (61.01)
12.56::0.15 (41.11) 11.33:10.05 (54.70)
26.03:10.04 (67.71)
Table 27: The performance (mean (%) + std (%) comparison of reprogrammed ViT-S+ConvStem on Food101, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
/ Out
Strategy (x x)
Strategy 2 (X / v)
Strategy 3 (V/) /x)
Strategy (V/v)
Pad/ IterLM
0.00 (16.23)
0.51 (0.99)
1.04=0010 (12.27)
4.30 (11.24)
Pad/ / FullyLM
0.00+1.00 (56.99)
4.36:10.05 (13.35)
12.29:10.17 (48.74)
21.23:10.02 (45.54)
Full IterLMM
0.00++0000 (15.62)
0.25++0.23 (1.71)
0.84=0.04 (12.50)
0.98++0.00 (1.00)
Full / FullyLL
0.00=1000 (67.94)
12.41=:0.05 (31.70)
10.91 (53.47)
23.15:10.07 (50.34)
Table 28:
The performance (mean (%) + std (%)) comparison of reprogrammed ResNet-18 on ChestX-Ray, where PGD-10 is used for adversarial training, and PGD-20 is used for adversarial testing. The results are presented in the format of robustness accuracy (clean accuracy), where the std of clean accuracy is omitted. The best robustness is highlighted in bold.
Adversarially Pre-trained Model VR using Adversarial Training
In Out /ut
Strategy 1 (X / X) Strategy v/)
Strategy (V/ / X)
Strategy (V/V)
Pad/ IterLM
0.00+1000 (70.31)
32.23 (56.30)
43.95=10233 (66.21)
49.41110.79 (64.26)
Pad/ / FullyLM
21.68=0.87 (72.07)
52.31 (64.23)
46.29-0.10 (71.68)
55.86-10.04 (70.51)
Full / TterLM
10.74=0.22 (74.80)
50.30+ (54.30)
33.59=0045 (67.77)
52.34+ (65.04)
Full FullyLM
(74.6))))
54.30=10.31 (66.79) 9.57=10.05 (71.48)
55.86 (71.88)
C.4 EXPERIMENTS UNDER DIFFERENT ADVERSARIAL ATTACKS
To further verify the effectiveness our strategy in a more comprehensive way, we verify the robustness under more adversarial attacks, including FGSM (Goodfellow et al. (0044). Square (An- driushchenko et al. 2020) and Attoct (Croce Hein 2020). The experimental results are These results show that different adversarial attack methods will bring different
shown in Table 29 adversarial robustness results, and our strategy can achieve satisfactory adversarial robustness on all these methods.
24

---

Published as a conference paper at ICLR 2025
Table 29: The performance (mean (%) std (%)) comparison of VR (Full FullyI. with ViT-
S+ConvStem (A.T.) on different datasets under different adversarial attacks. Using PGD-10 (c 4/255) adversarial training.
Dataset
Clean Acc.
FGSM
PGD-20
Square
AutoAttack
Flowers 102 OxfordPets
71.69:1.24
47.18
44.0
60.42.+0.03
40.89
83.04=0.18
54.3610.26
49.33=0.12
70.81
48.42 14
SUN397 Food101
67.17=3.49
38.50=0.61
26.03=0.04
55.95=0.69
30.02
50.34
24.8810.110
23.15
34.88
19.26:0.15
Strategy 1
Clean
Adv.
Strategy 2
Clean
Adv.
Strategy 3
Clean
Adv.
Strategy 4
Clean
Adv.
Figure 4: Visual results of reprogramming a ResNet-18 18 the CIFAR-10 dataset, where the visual input transformation used for reprogramming is Full and the output label mapping is FullyLM. For each group of images, the leftmost image is the original image from the target domain, the middle image shows the noise added by the visual input transformation, and the rightmost image is the output after applying the visual input transformation.
Strategy 1
Clean
Adv.
Strategy 2
Clean
Adv.
Strategy
Clean
Adv.
Strategy
Clean
Figure 5: Visual results of reprogramming a ResNet-18 the CIFAR-10 dataset, where the visual input transformation used for reprogramming is Pad and the output label mapping is FullyLM. For each group of images, the leftmost image is the original image from the target domain, the middle image shows the noise added by the visual input transformation, and the rightmost image is the output after applying the visual input transformation.
25


Reviews:
[{"review_id": "wmZhy5lOPm", "reviewer": "Reviewer_1Jug", "summary": "This paper delves into the adversarial robustness of visual reprogramming. It first highlights the adversarial vulnerability of existing VR methods and proposes to alleviate this issue with two strategies, adopting adversarial trained models and adversarial training in VR. A theoretical analysis is also provided to support the practices.", "questions": "1. Why $f_{out}$ is concatenated to $f_\\mathcal{S}$ in the last term of the equation in Definition 1 ? Is there any intuitive explanation on why the source domain $\\mathcal{S}$ should also be mapped to the $\\mathbb{R}^\\mathcal{T}$ ?\n\nI will raise my rating if the authors can properly addressed my concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The adversarial robustness of visual reprogramming is a critical issue.\n2. The proposed methods significantly improve the adversarial robustness.\n3. The theoretical upper bound of adversarial risks is straightforward yet valuable, validating the empricial findings.\n4. Multiple strategies of VR are considered in experiments.", "weaknesses": "1. The two proposed strategies are mature techniques in the field. Technically, it's simply an extension to visual reprogramming and there are no surprising results. Meanwhile, as the number of learnable parameters in visual reprogramming is limited, adversarial training may not be an ideal solution to improve the robustness.\n2. Although the theoretical analysis correponds to the proposed strategies, the AMDD-VRA part, which takes up the most context, is not reflected or applied in practice. This term seems to be the most important factor in VR, which is neither taken into consideration in methods nor discussed based on experimental results. For instance, the performance-robustness trade-off on GT-SRB is more noticeable than that on CIFAR. Does it has anything to do with the gap between the source domain and the target domain?\n3. There can be more discussions on the recent advancements in Visual Reprogramming and Visual Prompting. For instance, [1] has studied VP with adversarial robustness, what's the difference between your work and theirs. Also, [2] has extended VP to visual foundation models, is your work applicable to more significant scenarios like MLLMs?\n\nMinor: There are some typos in submission, e.g., 'iss' in line 191.\n\n[1] Chen, Aochuan, et al. \"Visual prompting for adversarial robustness.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.\n[2] Zhang, Yichi, et al. \"Exploring the Transferability of Visual Prompting for Multimodal Large Language Models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QbuRUctTnD", "reviewer": "Reviewer_xfee", "summary": "This paper focuses on the adversarial robustness of visual reprogramming (VR) against adversarial attacks. To this end, it first empirically finds that reprogramming pre-trained models with adversarial robustness and incorporating adversarial samples from the target task during reprogramming can both improve the adversarial robustness of reprogrammed models. Furthermore, it also presents a theoretically guaranteed adversarial robustness risk upper bound for VR. Experiments on several benchmarks validate the effectiveness of the proposed method.", "questions": "Please see the Weaknesses for details.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is interesting. The paper is well-written, and easy to understand.\n2. The authors introduced an upper bound for the adversarial robustness risk in visual reprogramming, theoretically proving the effectiveness of the proposed method.\n3. Extensive comparative experiments were conducted on multiple datasets, showcasing the performance of the proposed method across different attack iteration numbers and perturbation sizes.", "weaknesses": "1. The technical novelty of this paper is limited. The modules for visual programming, such as input transformation and label mapping, are similar to (Chen, 2022; Cai et al. 2024). And those modules for adversarial robustness also have been proposed by previous literature (Madry et al. 2018; Zhang et al., 2019a)\n2. I'm a little confused about the task setting, especially in the availability of adversarially pre-trained models. For the source domain, it may not be practical to have both a pre-trained model and an adversarially pre-trained model. Obviously, we can’t temporarily train a source model via adversarial training. if we can, why don't we train a target model directly with clean and adversarial data, because both datasets are also available?\n3. Experiments in Sec. 5 are more like ablation studies, to discuss the effectiveness of different designs for the proposed framework. However, it lacks comparisons with kinds of baselines or SOTA methods. Moreover, it would be more significant to evaluate the proposed method with more complex models on more large datasets, because that is the point of visual reprogramming to reduce the training and storage costs for target domains.\n4. In Eq. 10, $\\Phi_{\\rho}$ has two input variables (i.e., $x$ and $y$), but in the definition of Eq. 11, it only has one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yUVjhsWwtn", "reviewer": "Reviewer_krkJ", "summary": "The goal of Visual reprogramming is re-purposing pre-trained models for new tasks without retraining the model itself, and only learning a lightweight visual input transformation and label remapping of the output. Prior works have used different kinds of input transforms, remapping functions, losses etc, while this work focuses on the adversarial robustness aspect. The authors find that reprogramming naturally trained models results in almost zero robustness against adversarial attacks. Empirically, the paper demonstrates that using adversarially pre-trained models as the backbone for reprogramming and using adversarial training during the reprogramming both improve robustness. Combining both strategies (adversarially pre-trained backbones + adversarial training) yields the best results. The paper also provides some theoretical lemmas and proofs to establish an upper bound for adversarial risk of visual reprogramming.", "questions": "1. The authors could clarify more on how their theoretical work on guaranteed adversarial robustness risk upper bound affects their method and potentially what are the implications of their proof for this task\n\n2. Would these results generalize to more practical datasets such as ImageNet and recent models such as VisionTransformers, which have very different robustness characteristics than CNNs ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is easy to follow \n\n+ The work explores different combinations of input transformations and label remapping techniques, which strengthens confidence in the results.", "weaknesses": "- Limited novelty, the main component of the method is using adversarially pre-trained backbones, and using adversarial training during visual re-programming, which is a direct application of prior work.\n\n- The theoretical portion of the paper tries to tackle an interesting question, however the relevance to the method is limited ? The method itself is very empirical and intuitive, and the theory seems like an afterthought.\n\n- Though the authors demonstrate results on various input transform/label mapping techniques, in terms of backbones only ResNet18 is used, and only on CIFAR-10/100 and GTSRB datasets, which consist of small images, which is qualitatively different from practical image recognition tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sd3MiPoa60", "reviewer": "Reviewer_R93c", "summary": "This paper investigates the adversarial robustness of visual reprogramming (VR), a method that adapts pre-trained models for new tasks without fine-tuning. The authors find that existing VR methods are vulnerable to adversarial attacks, but can be significantly improved by using adversarially robust pre-trained models and incorporating adversarial examples from the target task during reprogramming. The paper presents a theoretical analysis that bounds the adversarial risk of VR and provides a foundation for future research. Experiments demonstrate the effectiveness of the proposed strategies in enhancing the robustness of reprogrammed models.", "questions": "The paper focuses on image classification tasks. VR can be applied to a wider range of tasks, such as object detection and semantic segmentation. Can the methods in this article be extended to other downstream tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Pioneering exploration of VR's adversarial robustness: This paper is the first to focus on the vulnerability of VR models when facing adversarial attacks and proposes corresponding solutions, filling a gap in this field.\n- The article not only experimentally verifies that using adversarial pre-trained models and adversarial samples can improve the adversarial robustness of VR models but also proposes a theoretical guarantee of an adversarial robustness risk upper bound, providing a theoretical foundation for future research.", "weaknesses": "- I am concerned about the application scenarios of the proposed method. Visual reprogramming aims to leverage less data and computational resource overhead to stimulate the base model's capabilities on target tasks, which means that timeliness and performance are more important than adversarial robustness. The methods proposed in the article hinder these two aspects to some extent.\n- The paper primarily uses the projected gradient descent (PGD) method to generate adversarial samples. However, different generation methods may have different impacts on the model's robustness, and there is a lack of discussion on more adversarial sample generation methods.\n- This work focuses on classification accuracy on adversarial samples, but adversarial robustness is a multidimensional concept that requires more comprehensive evaluation metrics. Adversarial training has some limitations, such as high computational costs and the risk of overfitting, which require more discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)