Pre-trained Adversarial Perturbations
Authors: Yuanhao Ban
Yinpeng Dong
Venue: NeurIPS 2022 Accept (2022)
Keywords: Adversarial samples
pre-trained models
security

Abstract: Self-supervised pre-training has drawn increasing attention in recent years due to its superior performance on numerous downstream tasks after fine-tuning. However, it is well-known that deep learning models lack the robustness to adversarial examples, which can also invoke security issues to pre-trained models, despite being less explored. In this paper, we delve into the robustness of pre-trained models by introducing Pre-trained Adversarial Perturbations (PAPs), which are universal perturbations crafted for the pre-trained models to maintain the effectiveness when attacking fine-tuned ones without any knowledge of the downstream tasks. To this end, we propose a Low-Level Layer Lifting Attack (L4A) method to generate effective PAPs by lifting the neuron activations of low-level layers of the pre-trained models. Equipped with an enhanced noise augmentation strategy, L4A is effective at generating more transferable PAPs against the fine-tuned models. Extensive experiments on typical pre-trained vision models and ten downstream tasks demonstrate that our method improves the attack success rate by a large margin compared to the state-of-the-art methods.

Pre-trained Adversarial Perturbations
Yuanhao Ban .2,, Yinpeng Dong 1,3t
Department of Computer Science Technology. Institute for AL, BNRist Center,
Tsinghua-Bose Bosshh Joint ML Center, THBI Lab, Tsinghua University
2 Department of Electronic Engineering, Tsinghua University 3 RealAI
banyh19@mails, hts.e/eee tsinghua. edu. en, dongyinpengdmail. seappengdaall..tnnnuaaaaa. edu. en
Abstract
Self-supervised pre-training has drawn increasing attention in recent years due its superior performance numerous downstream tasks after fine-tuning How- ever, it well-known that deep learning models lack the robustness to adversarial examples, which can also invoke security issues pre-trained models, despite be- ing less explored. this paper, we delve into the robustness of pre-trained models by introducing Pre-trained Adversarial Perturbations (PAPs), which universal perturbations crafted for the pre-trained models to maintain the effectiveness when attacking fine-tuned ones without any knowledge of the downstream tasks. To this end, we propose a LeweLevel Layer Lifting Attack (L4A) method to generate ef- fective PAPs by lifting the neuron activations of low-level layers of the pre-trained models. Equipped with an enhanced noise augmentation strategy. L4A is effective at generating more transferable PAPs against fine- munettned models. Extensive experi- ments on typical pre-trained vision models and ten downstream tasks demonstrate that our method improves the attack success rate by large margin compared with state-of-the-arr methods.
Introduction
Large-scale pre-trained models [40, 13] have recently achieved unprecedented success in a variety of fields, e.g., natural language processing [21, 27, 1], computer vision [2, 16, 17]. A large amount of work proposes sophisticated self-supervised learning algorithms, enabling the pre-trained models to extract useful knowledge from large-scale unlabeled datasets. The pre-trained models consequently facilitate downstream tasks through transfer learning or fine-tuning [37, 50, 12J. Nowadays, more practitioners without sufficient computational resources or training data tend fine-tune the publicly available pre-trained models on their own datasets. Therefore, it has become an emerging trend to adopt the paradigm pre-training to fine-tuning rather than training from scratch [13].
Despite the excellent performance of deep learning models, they are incredibly vulnerable to adver- sarial examples [44, 11], which are generated by adding small, human- imperccipppe perturbations to natural examples, but can make the target model output erroneous predictions. Adversarial examples also exhibit intriguing property called transferability [44, 26, 32], which means that the adversar- ial perturbations generated for one model or set of images can remain adversarial for others. For example, universal adversarial perturbation (UAP) [32] can be generated for the entire distribution of data samples, demonstrating excellent cross cross-data transferability. Other work [26, 7, 47, 8, 34] has revealed that adversarial examples have high cross-model and cross-domain transferability, making black-box attacks practical without any knowledge of the target model or even the training data. However, much less effort has been devoted exploring the adversarial robustness of tra-tra models. As these models have been broadly studied and deployed in various real-world applications, "This work was done when Yuanhao Ban was intern at RealAI, Inc; CCoressponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Lifting the activations of low- level layers
Wrong "Albutross"
Userl
"Flicker"
Fine- lumed model
Attacker
PAP
Pre-trained model
Wrong answer "Hamster"
Unidorm
Gaussian
Sampling
noise
User2
"Truck"
Fine-tuned model
Figure 1: A demonstration of pre-trained adversarial perturbations (PAPs): An attacker first down- loads pre-trained weights on the Internet and generates PAP by lifting the neuron activations of low-level layers of the pre-trained models. We adopt a data augmentation technique called uniform Gaussian sampling to improve the transferability of PAP. When users fine-tune the pre-trained mod- els to complete downstream tasks, the attacker can add the PAP to the input of the fine-tuned models to cheat them without knowing the specific downstream tasks.
it is of significant importance to identify their weaknesses and evaluate their robustness, especially concerning the pre-training to the fine-tuning procedure.
In this paper, introduce Pre-trained Adversarial Perturbations (PAPs), new kind of universal adversarial perturbations designed for pre-trained models. Specifically, a PAP is generated for pre- trained model to effectively fool any downstream model obtained by fine-tuning the pre- tre-traine one, as illustrated in Fig. dataset, and fine-tuned model parameters are all unavailable. This attack setting more suitable for the pre-training to the fine-tuning procedure since many pre-trained models are publicly available, and the adversary may generate PAPs before the pre-trained model has been fine-tuned. Although there are many methods [7, 47] proposed for improving the transferability. they do not consider the specific characteristics of the pre-training to the fine-tuning procedure, limiting their rross-fnetuning transferability in our setting.
works under a quasi-black-box setting where the downstream task,
To generate more effective PAPs, we propose Low-Level Layer Lifting Attack (I4A) method, which aims to lift the feature activations of low-level layers. Motivated by the finding that the lower the level of the model's layer is, the less its parameters change during fine-tuning, to destroy the low-level feature representations pre-traine trained models, making the attacking effeets better reserved after fine-tuning. To further alleviate the overfitting of PAPs to the source domain, we improve L4A with a noise augmentation technique. We conduct extensive experiments on typical pre-trained vision models [2, 17] and ten downstream tasks. The evaluation results demonstrate that our method achieves a higher attack success rate on average compared with the alternative baselines.
generate PAPs
2 Related work
Self-supervised learning. Self-supervised learning (SSL) enables learning from unlabeled data. To achieve this, early approaches utilize hand-crafted pretext tasks, including colorization [53], rotation prediction [10], position prediction [36], and Selfie [45]. Another approach for SSL is contrastive learning [25, 39, 2, 22], which aims to map the input image to the feature space and minimize the distance between similar ones while keeping dissimilar ones far away from each other. In particular, a similar sample is retrieved by applying appropriate data augmentation techniques to the original one, and the versions of different samples viewed as dissimilar pairs.
Adversarial examples. With the knowledge of the structure and parameters of model, many algorithms [24, 31, 30, 38] successfully fool the target model in white-box manner. An intriguing property of adversarial examples is their good transferability [26, 32]. The universal adversarial perturbations [32] demonstrate good cross-data tranffcrability by optimizing under distribution of

---

data samples. The cross- moos- transferability has also been extensively studied [7, 47, 8), enabling the attack on black-box models without any knowledge of their internal working mechanisms.
Robustness of the pre-training to fine-tuned procedure. Due to the popularity of pre-trained models, a lot of works [43, 49, 4] study the robustness of this setting. Among them, Dong al. [6] propose novel adversarial fine-tuning method in an features learned from the pre-trained model. Jiang et al. [20] integrate adversarial samples into the pre-training procedure to defend against attacks. Fan et at. [9] adopt Clusterfit [48] to generate pseudo-labe label data and later use them for training the model in supervised way, which improves the robustness of the pre-train pre-trained model. The main difference between our work and theirs is that we consider the problem from an attacker's perspective.
way to retain robust
3 Methodology
In this section, we first introduce the notations and the problem formulation of the Pre-trained Ad- versarial Perturbations (PAPs). Then, we detail the Low-Level Layer Lifting Attack (L4A) method.
3.1 Notations and problem formulation
Let fe denote a pre-trained model for feature extraction with parameters O. It takes an image x e Dp as input and outputs a feature vector v E X, where D, and X' refer to the pre-training dataset and feature space, respectively. We denote fx(x (x) as the k-th layer's feature map of fe for an input image x. In the pre-training to fine-tuning paradigm, user fine-tuner the pre-trained model fo using a new dataset D, of the downstream task and finally gets a fine-tuned model for with updated parameters 0'. Then, let for (x) be the predicted probability distribution of an image x over the classes of D,, and For xx) = arg max So so((x) (x) be the final classification result.
In this paper, we introduce Pre-trained Adversarial Perturbations (PAPs), which are generated for the pre-trained model fe. but can effectively fool fine-tuned models for on downstream tasks. Formally, a PAP is universal perturbation 6 within a small budget crafted by fe and Dp, such that [x + 6) j Fer (x) for most of the instances belonging to the fine-tuning dataset Dz. This can be formulated as the following optimization problem:
max Ex-D, [Fo (x) + For (x + 6)], s.t. 1|01| S and x + 8 E 1],
(I)
where ||
|| denotes the lp norm and we take the fox norm this work. There exist some works related to the universal perturbations, such as the universal adversarial perturbation (UAP) [32] and the fast feature fool (FFF) [33], as detailed below.
UAP: Given a classifier ʃ and its dataset D, the UAP tries to generate a perturbation ò that can fool the model on most of the instances from D, which is usually solved by an iterative method, Every time sampling an image x from the dataset DD the attacker computes the minimal perturbation Ç that sends + 5 to the decision boundary by Eq. (2) and then adds into 8.
Ç + arg min ||r]|,, s.t. F(x 8 +r) + F(x).
FFF: It aims to produce maximal spurious activations at each layer. To achieve this, FFF starts with random 8 and solves the following problem:
logg (11 s.t. ||01|p Sc.
min 5
(3)
where T,(5) is the mean of the output tensor at layer i.
3.2 Our design
However, these attacks show limited cross- cross-finetuingg transferability in our problem setting due to ignorance of the fine-tuning procedure. Two challenges are degenerating the performance.
Fine-tuning Deviation. The parameters of the model could change lot during fine-tuning. As a result, the generated adversarial samples may perform well the feature space of the pre-trained model but fail in the fine-tuned ones.
3

---

CME
CAR
Releiive Changing Bele
DRD
Selaivv Changing
Releiive Changing
0.8
0.6
CHAITE
CHARSEC
0.0
OIAATEO
CUS DAn
02
0.0
Resnet50
(b) Resnet101
(c) VITI6
Figure 2: The ordinate represents the Frobenius norm of the difference between the parameters of the fine-tuned model and its corresponding pre-trained model, which is scaled into a range from O 1 for easy comparison. The abscissa represents the level of the layer. Note that Resnet50 and Resnet101 [14] are pre-trained by SimCLRv2 [2], and VITI6 [46] is pre-trained by MAE [17].
Datasets Deviation. The statisties (i.e., mean and standard deviation) of different datasets can vary lot. Only using the pre-training dataset with the fixed statisties to generate adversarial samples may suffer a performance drop.
To alleviate the negative effect of the above issues, we propose a Low-Level Layer Lifting Attack (L4A) method equipped with uniform Gaussian sampling strategy.
Low-Level Layer Lifting Attack (L4A). Our method is motivated by the findings in Fig. that the higher the level of the layers, the more their parameters change during fine-tuning" This is also consistent with the knowledge that the low-level convolutional layer acts as an edge detector that extracts low-level features like edges and textures and has little high-level semantic information [37, 50]. Since images from different datasets share the same low-level features, the parameters of these layers can be preserved during fine-tuning. In contrast, the attack algorithms based the high-level layers or the scores predicted by the model may not transfer well such a cross cross-finetune finetfninttnngggg setting, as the feature spaces of high-level layers are easily distorted during fine-tuning. The basic method of L4A can be formulated as the following problem:
min Lhose (Jo, x, 6) -Ex-D
| 8)12
where || || denotes the Frobenius norm of the input tensor. In our experiments, we find the lower the layer, the better it performs, so we choose the first layer as default, such that k 1. As Eg. (4) usually a sophisticated non-convex optimization problem, we solve it using stochastic gradient descent method.
We also find that fusing the adversarial loss of the consecutive low-level layers boost the perfor- mance, which gives L4Arue method as solving:
min Luse (fo, 0) = -X |150 (x 8) + x 11 + 80112
(5)
where f (x+d) and /b: (x+ + 8) refers to the -th and 1iz layers' feature maps of fo respectively, is a balancing hyperparameter. We set ki = 1 and Ai as default.
Uniform Gaussian Sampling. Nowadays, most state-of-the-art networks apply batch normalization [19] input images for bet- ter performance. Thus, the datasets' statistics become an essen- tial factor for training. As shown in Fig. the distribution of the downstream datasets can vary significantly compared to that of the pre-trate pre-training dataset. However, traditional data augmentation techniques [51, 18] are limited to the pre-training domain and cannot alleviate the problem. Thus, we propose sampling Gaus- sian noises with various means and deviations to avoid overfit- ting. Combining the base loss using the pre-training dataset and the new loss using uniform Gaussian noises gives the 1.4ugs
0.28
Pood
Standantt Desiation
DTD
DRAATO
rove
0.24
cuB
Protro ned Datoser
S1L10
Imagenct
0.20
0.64
0.46
0.18
0.50
6.52
Mean
Figure 3: Datasets' statistics.

---

Table 1: The attack success (%) of various attack methods against Resnet101 pre-trained by
SimCLRv2. Note that C10 stands for CIFARIO and C100 stands for CIFAR100.
ASR
Cars
Pets
Food
DTD
FGVC
CUB
SVHN
C10
C100
STL10
AVG
FFF
43.81
38.62
49.95
63.24
85.57
48.38
12.55
8.53
77.74
57.11
48,55
FFFF
33.93
31.37
41.77
52.66
78.94
45.00
14.85
14.42
72.59
56.66
44.22
FFF
31.87
29.74
39.25
46.92
74.17
43.87
9.24
11.77
65.61
50.21
40.26
DR
36.28
35.54
47.43
47.45
75.00
44.15
12.05
21.35
65.39
41.65
42.63
SSP
32.89
30.50
43.12
45.85
82.57
45.55
8.69
11.66
65.80
40.91
40.75
ASV
60.75
19.84
36.33
56.22
84.16
55.82
7.11
7.29
58.10
80.89
46.64
UAP
48.70
36.55
60.80
63.40
76.06
52.64
8.46
8.53
52.35
31.15
43.86
UAPEPGD
94.12
66.66
61,30
72.55
70.34
82.72
13.88
61.65
20.04
50.13
59.34
L4A
94.07
61.57
71.23
69.20
96.28
81.07
11.70
12.68
80.57
90.49
66.89
1,
90.98
88.53
80.65
74.31
93.79
91.23
11.40
17.40
80.98
89.69
67.10
L4Aug
94.24
94.99
78.28
77.23
92.92
91.77
11.40
14.60
76.50
90.05
72.20
method follows:
[1Lb(x 8)1 + 1158 + 8))1
min Lugs x. -B Ex 5
(6)
where a and are drawn from the uniform distribution (n) and on), respectively, and My Pa or are four hyperparameters.
Experiments
We provide some experimental results in this section. More results can found in Appendix. Our code is publicly available at https: ////tttttbb//////////// com/banyuanhao/PA.
4.1 Settings
Pre-training methods. SimCLR [2, 3] uses the Resnet [14] backbone and pre-trains the model by contrastive learning. We download pre-trained parameters of Resnet50 and Resnet101 evaluate the generalization ability of our algorithm different architectures. We also adopt MOCO [15] with the backbone of Resnet502. Besides convolutional neural networks, transformers [46] attract much attention nowadays for their competitive performance. Based on transformers and masked image modeling, MAE [17] becomes good alternative for pre-training. We adopt the pre-trained ViT-base-1e 16 model?. Moreover, vision-language pre- pre-trained models gaining popularity these days. Thus we also choose CLIP [41]* for our study. We report the results of SimCLR and MAE in Section 4.2. More results CL.P and MOCO can be found in Appendix A.1.
Datasets and Pre-processing. We adopt the ILSVRC 2012 dataset [42] to generate PAPs, which are also used to pre-train the models. We mainly evaluate PAPs on image classification tasks, which are the same as the settings of SimCLRv2. Ten fine-grained and coarse-grained datasets are used to test the cross-finetie wross-finetunin transferability of the generated PAPs. We load these datasets from torchvision (Details A)pendixD). Before feeding the images to the model, we resize them to 256 x 256 and then center crop them into 224 x 224.
Compared methods. We choose UAP [32] to test whether image-agnostic attacks also bear good cross-finetuniee transferability. Since UAP needs final classification predictions of the inputs, we fit a linear head on the pre-trained feature extractor. Furthermore, by integrating the moment term into the iterative method, UAPEPGI [5] is believed to enhance cross-model transferability. Thus, we adopt UAPEPGD to study the connection between cross-min moosll and cross-finetuning transferability. As our algorithm is based on the feature level, other feature attacks (including FFF [33], ASV [23], DR [28], SSP [35]) are chosen for comparison.
httpp:///gtttabb.o//o////////// http:///gghhaab.oooooogeeeeeerrr chmsimclr
htt////..haippuliccieleeee.. wtt.e//wssls.ccccoooo/oeooo com/noco/
5

---

Table 2: The attack success rate (%) of various attack methods against Resnet50 pre-trained by SimCLRv2. Note that C10 stands for CIFARIO, and C100 stands for CIFARI00.
ASR
Cars
Pets
Food
DTD
FGVC
CUB
SVHIN
C10
C100
STL10
AVG
FFF no
26.91
30.83
43.28
48.99
41.30
38.23
79.00
68.50
44.67
16.95
43.86
FFFmean
36.75
33.88
45.26
50.15
53.13
77.22
52.02
82.41
68.10
22.11
52.10
FFF
37.88
35.30
52.79
59.52
59.62
57.04
80.33
75.40
53.58
18.31
52.98
DR
38.64
34.42
50.04
45.53
39.80
75.67
47.88
76.05
60.57
13.98
48.26
SSP
41.70
43.94
50.83
48.78
47.67
82.39
48.38
86.95
66.23
19.56
53.64
ASV
74.47
36.93
45.85
73.51
64.89
92.29
73.16
45.14
53.60
22.02
58.19
UAP
44.86
46.47
64.67
65.53
49.63
82.32
52.00
79.63
46.46
19.99
55.16
UAPEPGD
66.29
66,58
81.11
69.52
87.91
59.07
69.16
87.84
68.26
37.12
69,28
LLLLbsse
94.86
56,30
61.31
75.37
67.61
94.87
81.45
68.25
77.04
34.56
66.89
L4Arss
96.00
59.80
65.00
77.93
69.39
95.02
85.05
64.41
76.29
37.54
72.64
L4Aug
96.13
79.15
74.87
82.18
78.73
94.45
95.29
55.03
77.10
45.09
77.80
Table The attack success rate (%) of various attack methods against ViT16 pre-trained by MAE. Note that C10 stands for CIFARIO and C100 stands for CIFAR 100.
ASR
Cars
Pcts
Food
DTD
FGVC
CUB
SVHN
CTO
C100
STL10
AVG
FFF
64.31
88.21
95.04
88.18
81.91
92.94
76.10
49.48
79.83
60.91
77.69
FFF
40.39
67.54
54.10
61.38
71.47
73.39
92.96
86.88
94.55
67.90
71.06
FFF
48.36
77.89
60.06
64.04
75.67
74.09
92.33
86.13
94.48
70.40
74.35
DR
37.02
23.84
59.54
44.73
28.01
10.41
14.30
16.66
14.12
21.54
27.02
SSP
44.15
73.31
85.42
72.82
52.57
63.10
52.45
27.94
25.32
36.90
53.40
ASV
38.46
10.17
37.49
48.31
29.14
4.97
8.41
17.04
11.34
21.14
22.64
UAP
62.71
58.90
89,90
74.92
44.69
39.56
47.65
47.77
33.80
51.70
55.16
UAPEPGD
63.67
73.09
96.22
76.69
57.78
73.37
79.84
45.89
47.21
55.79
66.95
L4Apus
87.66
89,98
98.96
99.10
99.33
84,06
86.99
98.62
97.08
98.25
94,00
L4Afue
83.24
89.57
98.87
98.77
98.36
93.60
89.85
98.64
95.72
97.53
94.42
LAAngs
96.49
90.00
98.97
98.89
99.48
84.01
89.56
99.43
97.27
98.96
95.30
Default settings and Metric. Unless otherwise specified, choose a batch size of 16 and step size of 0.0002. All the perturbations should be within the bound of 0.05 under the fo norm. We evaluate the perturbations the iterations of 00000 5, 000, 30, 000, and 60, 000, and report the best performance. We show the results in the measure of attack success rates (ASR), representing the classification error rate on the whole testing dataset after adding the perturbations the legitimate images.
4.2 Main results
We craft pre-trained adversarial perturbations (PAPs) for three pre-trained models (i.e., Resnet50 by SimCLRv2, Resnet101 by SimCLRv2, VITI6 by MAE) and evaluate the attack success rates on ten downstream tasks. The results are shown in Table Table 2, and Table 3, respectively. Note that the first seven datasets are fine-grained, while the last three are yoarse-graines ones. We mark the best results for each dataset in bold, and the best baseline in blue. We highlight the results of L4Augs in red to emphasize that the L4A attack equipped with Uniform Gaussian Sampling shows great trass-finetuming transferability and performs best.
quick glimpse shows that our proposed methods outperform all the baselines by a large margin. For example, as can be seen from Table I, if the target model is Resnet101 pre-trained by SimCLRv2. the best competitor FFF mean achieves an average attack success rate of 59.34%, while the villain L4A can lift up to 66.89% and the UGS technique further boosts the performance up to 72.20% %. Moreover, the STL10 dataset is the hardest for PAPs to transfer among these tasks. However, L4Aug- can significantly improve the cross cross-finetune finetuning transferability. achieving an attack success rate of 90.05% and 98. 98.96% for Resnet101 and VIT16 in STL10, respectively. Another intriguing finding is that ViT16 with a transformer backbone shows severe vulnerabilities to PAPs. Although performing best on legitimate samples, they bear attack success rate of 95.30 % under the L4A.gs attack, closing to random outputs. These results reveal the serious security problem of the pre-training to fine-tuning paradigm and demonstrate the effectiveness of our method in such problem setting.

---

100
94.55
92.96 7..6
Attack Success Rate
Attack Success Rate
80
85.67
82.62
80
77,31
12.07
71.83
67.1
62.07
60
20
60
40
IVG
10
AVG
Resnet50
Resnet50
20
Resnet100
20
Ressrel101
MTI6
VIT16
layeri
layer
blockO
layer
layer
blocks
layera layera
block6
layers layers blocks
layers owerr bioze1
layeri
layers
layer2 1ock
layer2
bloc3
1aye13
layer3
blckss
layer4
leyeré
block9
lavers layera block]l
Figure 4: The attack success rates (%) of L4Apase when using different layers. We show the results on the pre-training domain (Left) and fine-tuning domains (Right).
100
100
Alair Succoss Bate
60
OPARO
40
40
Ather
2
10
20 10 14
20
10
10
10" 10-15 10th
10
10
20
10
10
10
10
10
10
20
10th
1015
(a)
Resnet101
(b) Resnet50
(c)
VITI6
Figure 5: The attack success rate (%) of different hyperparameter L4A for different models.
4.3
Ablation studies
4.3.1 Effect of the attacking layer
We analyze the influence of attacking different intermediate layers of the networks the perfor- mance of our proposed 1.Absse in the pre-training domain (ImageNet) and the fine-tuning domains (ten downstream tasks). To this end, divide Resnet50, Resnet101, and ViT into five blocks (De- tails in Appendix F.I) and conduct our algorithm on them. Note that for the fine-tuning domains, the average attack success rates the ten datasets are reported.
As shown Fig. 4, the lower the level we choose to attack, the better our algorithm performs in the fine-tuning domains. Moreover, for the pre-training domain, attacking the middle layers of the networks results a higher attack success rate compared the top and bottom layers, which also reported in existing works [29, 35, 52]. These results reveal the intrinsic property of the pre-training fine-tuning paradigm.
the lower-level layers change less during the fine-tuning procedure, attacking the low-level layer becomes more effective when generating adversarial perturbations in the pre-training domain rather than the middle-level layers.
4.3.2 Effect of uniform Gaussian sampling
We set μ4 Mi ot a 0.4, 0.6, 0.05, 0.1 for all the three models as they perform best. To discuss the effect of the hyperparameter
Eq. (6) fusing the base loss and the UGS loss, we select the values with grid of logarithmically spaced learning rates between 10 and 102. The results shown in Fig. 5. As shown in Fig. 5(a), the best attack success rate is achieved when 10-0.5 on Resnet101, boosting the performance by 1.52% compared to only using the Gaussian noises.
Furthermore, we study whether adopting the fixed statistics of the pre- pre-training dataset (i.e., the mean and report
Table 4: Fixed datasets' statistics
Model
R101
R50
MAE
standard deviation of ImageNet) can help. the attack success rates (%) in the Table 4, where None refers to using no data augmentation, ImageNet adopts the mean and standard deviation of ImageNet
None
66.95
71.16
94,00
ImageNet
68.39
69.50
95.30
Uniform
72.20
77.80
95.30

---

and Uniform samples pair of mean and standard deviation from the uniform distribution. As can be seen from the table, Uniform outperforms None by 4.39% on average. while ImageNet does not help, which means that our proposed UGS helps to avoid overfitring the pre-training domain.
4.4 Visualization of feature maps
We show the feature maps before and after L4Afse attack Fig. 6. The Left column shows the imputs of the model, while the Middle and the Right show the feature map of the pre-trained model and the fine-tuned one, respectively. The Upper row represents the pipeline of a clean input in Cars, and the Lower shows that of adversarial ones. We can see from the upper row that fine-tuning the model can make it sensitive the defining features related to the spe- cific domain, such as tires and lamps. However, adding an adversarial perturbation to the image can significantly lift all the activations and finally mask the useful features. Moreover, the effect of our attack could be well preserved during fine-tuning and cheat the fine-tuned model into misclassification, stressing the safety problem of pre-trained models.
Figure 6: Visualization of feature maps.
4.5 Trade-off between the clean accuracy and robustness
We study the effect of fine-tuning epochs on the performance of our attack. To this end, we fine-tune the model until it reports the best result on the testing dataset, and then we plot the clean accuracy and the ac- euracy against FGSM and PAPs Pets and STL10 in Fig. 7. The figure shows the clean accuracy and robustness of the fine-tuned model against PAPs are at odds. In Fig. 7(b), the model shows the best ro- bustness at epoch 5 in STL10, achieving an accuracy rate of 95.38% and 55.02% on clean and adversarial samples, respec-
100
80
08.51
95.33
97.07
PAP
60
FGSM
82
32.18
Clean
28.95
PSP
10.15
12.4b
RGSM
10
20
30
40
0
8
10
12
14 16 18
epochs
epochs
(a)
Pets
(b) STL.10
Figure Model accuracy (%) on the Pets and STL10 datasets under clean inputs, PAP, and FGSM attack.
tively. However, the model does not converge until epoch 19. Though the process boosts the clean accuracy by 1.63 %, it suffers a significant drop in robustness, as the accuracy on adversarial sam- ples lowered 28.96% %. Such findings reveal the safety problem of the pre-training to fine-tuning paradigm.
Discussion
In this section, we first introduce the gradient alignment and then use it explain the effectiveness of our method. In particular, we show why our algorithms fall behind UAPs in the pre-training domain but have better cross-finetuning transferability when evaluated the downstream tasks.
5.1 Preliminaries
Gradient sequences. Given network fe, and sample sequence {x1 X2, Xy...., XN to drawn from the dataset D, let A = ( Ade.. Ade..x. Ada. }'r be the sequence of gradients obtained when generating adversarial samples by the following equation:
Ad0 VaL(fo,, x,, 6:), with 6, = - + A
where T. denotes the loss function of iterative attack methods like UAP, FFR, and L4A.
Definition (Gradient alignment). Given a dataset D and a model fo,- the gradient alignment Gf an attack algorithm is defined as the expectation over the cosine similarity of Ad0 xi and Ade..x..
8

---

which can be formulated as Eq. (8)
Ad 0o.xx
GA Ex D
(8)
| 1 x
1
where Ad and x are two consecutive elements obtained by Eq. (7).
Then the 1.4 algorithm bears higher gradient alignment (A strict definition and proof in weaker form can be found AppendixB.] In addition, we provide the results of simulation experiment to justify it in Table 5. We can negative correlation between the gradient alignment and the attack success rate on ImageNet. In contrast, a positive correlation exists between the gradient alignment and attack success rate in the fine-tuning domain.
Effectiveness of the algorithm. Given a pre-trained model fe and the pre-training dataset Dp. the generation of PAPs can be reformulated from Eq. (1) to the following equation:
max 8E Span Dpo.pD)
Bx |Fer (x) + Fer (x + 8)}, s.t. ||0|| S and x + 8 8 (0,1],
where Span Span{Ddg.2) A denotes the subspace spanned by the elements of ~ôg P. The optimal value can be viewed as a linear combination of the elements in Adg.D,, so it is the feasible region of the equation. In particular, the feasible region of Eq. (9) is smaller than that of Eq. (1), which reflects that the stochastic gradient descent methods may not converge to the global maximum when is
not included in Spen A00.D,, and the local maximum of Eq. (9) in Span Adg. represents the effectiveness of the algorithm the fine-tuning domain.
Pre-training Domain
Fine-tuning Domain
Resnet101
GA
ImageNet
AVG
FFF no
0.1221
38.64%
48.55%
Decision boundary of xp
Decision boundary of
FFF mean
0.0194
48.37%
44.22%
FFF
0.1222
34.15%
40.26%
66gx:
DR
0.0165
41.06%
42.62%
UAP
0.0018
88.14%
43.86%
Fail!
UAPEPGD
0.0008
94.62%
59.39%
AB A8gxp 8xp
Xpp
Xt
SSP
0.0274
41.81%
40.75%
8 Dabgay Sueceed!
L4A
0.6125
37.44%
66.95% %
Bdox
A8g,8t
Gradient alignments: Mid- dle: Attack success rates on the pre-training dataset; Right: Average attack success rates on downstream tasks. See more details in Ap- pendixB.2
Table 5: Left:
Gradient aligament of LAA costf) Grndient alignment couf"" costff po cooff"y yu
Figure& 8: Illustration
5.2 Explanation
We aim to explain why the effectiveness of our algorithm is better in the fine-tuning domain and worse in the pre-traine pre-training domain, as seen from Table Let Adø,D, and Δ0),D, be the feasible zones of L4A and UAP obtained by feeding instances from D, into the pre-trained model fo, re- spectively. Similarly, we can define Ado,D, and 26 9. D. Meanwhile, denote 8 and 8 0288, as the maxima in the pre- tre-training domain obtained by L4A and UAP respectively. An illustration is shown in Fig. 8, supposing there is only one step in the iterative method.
Pre-training domain: Because the gradients of UAP obtained by Eq. (2) represent the directions to the closest points on the decision boundary in the pre-training domain. Thus, limiting the feasible zone to 2d does little harm to the performance when evaluated in the pre-training domain. Meanwhile, according to the optimal objective, L4A finds the next best directions which are worse than those of UAPs. Thus, 8 36..0, performs better than ò 866..,, the pre-training domain.
Fine-tuning domain: According to the fact that UAP bears a low gradient alignment near to 0, the subspace spanned by the tensors in A6',.. is almost orthogonal to that spanned by the tensors in Ad% e which represent the best directions that send the sample to the decision boundary in the fine-tuning domain. Thus, limiting the feasible of Eq. (1) A8, suffers great drop in ASR when evaluated on the downstream tasks in Eq. (9). However, as shown in Table 5,

---

algorithm can achieve gradient alignment of up to 0.6125, which means that there is considerable overlap in the next best feasible region of D, in the pre-training domain obtained by L4A and that of 'Dy in the fine-tuning domain. Thus the performance of the best solution in Ado.D, D, is close to that of Ld which represents the next best solution in the fine-tuning domain. Finally we have Se. n, performs better than 6 D' in the fine-tuning domain.
In conclusion, the high gradient alignment guarantees high cross-finetuning transferabilityy
Societal impact
A potential negative societal impact of L4A is that malicious adversaries could use to cause se- curity/safety issues in real-world applications. As more people focus on the pre-trained models because of their excellent performance, fine-tuning pre-trained models provided by the cloud server becomes a panacea for deep learning practitioners. such settings, PAPs become a significant secu- rity flaw-as one can easily access the prototype pre-trained models and perform attacking algorithms on them. Our work appeals to big companies to delve further into the safety problem related to the vulnerability of pre-trained models.
7 Conclusion
In this paper, we address the safety problem of pre-trained models. In particular, attacker can use them to generate so-called pre-trained adversarial perturbations, achieving a high success rate on fine-tuned models without knowing the victim model and the specific downstream tasks. Consid- cring the inner qualities of the pre-training to fine-tuning paradigm, we propose a novel algorithm, L4A, which performs well in such problem settings. A limitation of L4A is that it performs worse than UAPs in the pre-training domain; we hope some upcoming work can fill the gap. Furthermore, L4A only utilizes the information in the pre-training domain. When the attacker obtains some infor- mation about the downstream tasks, like several unlabeled instances in the fine-tuning domain, he may be able enhance PAPs using the knowledge and further exacerbate the situation, which leave to future work. Thus, we hope our work can draw attention to the safety problem of pre-trained models to guarantee security.
trained
Acknowledgement
This work was supported by the National Key Research and Development Program of China (200AAAA0106000, 2020AAA0104304, 2020AAA0106302), NSFC Projects (Nos. 62061136001, 62076145, 62076147, U19B2034, U1811461 U19A2081, 61972224), Beijing NSF Project (No. IQ19016), BNRist (BNR2022CC0006), Tsinghua Institute for Guo Qiang, and the High Perfor- mance Computing Center, Tsinghua University. Y. Dong was also supported by the China National Postdoctoral Program for Innovative Talents and Shuimu Tsinghua Scholar Program.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language mod- els are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), pages 1877-1901, 2020.
[2] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1597- 2020.
[3] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. In Advances in Neural Infor- mation Processing Systems (NeurIPS), pages 22243-22255 2020.
[4] Tong Chen and Zhan Ma. Towards robust neural image compression: Adversarial attack and model finetuning. arXiv preprint 2121v:211..08611.2 2021.
10

---

[5] Yingpeng Deng and Lina Karam. Universal adversarial attack via enhanced projected gradi- descent. In IEEE International Conference on Image Processing (ICIP), pages 1241-1245, 2020.
[6] Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang. How should pre-trained language models be fine-tuned towards adversarial robustness? In Advances in Neural Information Processing Systems (NeurIPS), pages 4356-4369, 2021.
[7] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9185-9192 2018
[8] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses transferable adver- sarial examples by translation-invarian attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4312 4321, 2019.
[9] Lijie Fan, Sijia Liu, Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? In Advances Neural Information Processing Systems (NeurIPS), pages 21480-211 1480-21492, 2021.
[10] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learn- ing by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
[11] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. International Conference on Learning Representations (ICLR), 2015.
[12] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Roge- rio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4805- 4814, 2019.
[13] Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. Pre-trained models: Past, present and future. Al Open, pages 225-250 2021.
[14] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 770-778, 2016.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729 2020.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2020.
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, 2022.
[18] Hiroshi Inoue Data augmentation by pairing samples for images classification. arXiv preprint AXXi::1001.02299 2018.
[19] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learning (ICML), pages 448-4566 2015.
[20] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adver- sarial contrastive learning. In Advances Neural Information Processing Systems (NeurIPS). pages 11119-162200 2020.
11

---

[21] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL.), pages 4171- 4186, 2019.
[22] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems (NeurlPS), pages 18661-18673, 2020.
[23] Valentin Khrulkov and Ivan Oseledets. Art of singular vectors and universal adversarial pertur- bations. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages
2018.
[24] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security, pages 99-112. 2018.
[25] Phuc H Le-KKaa, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, pages 193907-193934, 2020.
[26] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations (ICLR), 2017.
[27] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint
2019.
[28] Yantao Lu, Yunhan Jia, Jianyu Wang, Bai Li, Weiheng Chai, Lawrence Carin, and Senem Veli- pasalar. Enhancing cross-task black-box transferability of adversarial examples with disper- sion reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 940-949 2020.
[29] Yantao Lu. Yunhan Jia, Jianyu Wang. Bai Li, Weiheng Chai, Lawrence Carin, and Senem Veli- pasalar. Enhancing cross-task black-box transferability of adversarial examples with disper- sion reduction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 940-949,75 2020.
[30] Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Confer- ence on Learning Representations (ICLR), 2018
[31] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 2574 2582, 2016.
[32] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Uni- versal adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1775-1773, 2017.
[33] Konda Reddy Mopuri, Utsav Garg, and R Venkatesh Babu. Fast feature fool: A data indepen- dent approach to universal adversarial perturbations. In British Machine Vision Conference (BMVC),2011
[34] Muhammad Muzammal Naseer, Salman H Khan, Muhammad Haris Khan, Fahad Shah- baz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. In Advances in Neural Information Processing Systems (NeurIPS), pages 12905-12915, 2019.
[35] Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self-supervised approach for adversarial robustness. In Proceedings of the IEEE/VV Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 262-271, 2020.
[36] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision (ECCV), pages 69-84, 2016.
12

---

[37] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid- level image representations using convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1717-1724, 2014.
[38] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Anan- thram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pages 372-387, 2016.
[39] Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In European Conference on Computer Vision (ECCV), pages 319-345, 2020.
[40] Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre- trained models for natural language processing: A survey. Science China Technological Sci- ences, pages 1872-1897, 2020.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine 2021.
Learning (ICML), pages
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision (IJCV), pages 21 2122 2015.
[43] Ananya B Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M Khapra. Im- proving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. Transactions of the Association for Computational Linguistics (TACL), pages 810-827, 2020.
[44] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
[45] Trieu H Trinh, Minh-Thang Thang Luong, and Quoe V Le. Selfie: Self-supervised pretraining for 02940, image embedding. arXiv preprint arXiv: 1906.02900 2019.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural informa- tion processing systems (NeurIPS), pages 6000-6010, 2017.
[47] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 2730-2739, 2019.
[48] Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clus- terfit: Improving generalization of visual representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4509-6518, 2020.
[49] Zonghan Yang and Yang Liu. On robust prefix-tuning for text classification. In International Conference on Learning Representations (ICLR), 2022.
[50] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems (NeurIPS), pages
2014.
[51] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6023 6032, 2019.
[52] Qilong Zhang, Xiaodan Li, Yuefeng Chen, Jingkuan Song, Lianli Gao, Yuan He, and Hui Xue. Beyond imagenet attack: Towards crafting adversarial examples for black-box domains. In International Conference Learning Representations (ICLR), 2021.
13

---

[53] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European
Conference on Computer Vision (ECCV), pages 649-666, 2016.
14


Reviews:
[{"review_id": "vyPyf_CxS9d", "reviewer": "Reviewer_ZvUM", "summary": "This paper deals with robustness of pretrained models (i.e. self-supervised feature learners which can be used for multiple downstream tasks).  The goal of this paper is to develop \"pretrained adversarial perturbations\" (short: PAP) -- perturbations that are crafted to attack pretrained models AND their downstream tasks, without having any knowledge about the downstream tasks.  To generate PAPs, a method is developed -- Low-Level Layer Lifting Attack (L4A), which \"lifts\" the neuron activations of lower-level layers of the neural network.  \n\nExperiments are performed on the downstream task of image classification.  Two pretraining methods are used:  SimCLR (with convolutional networks) and MAE (with transformer networks).  ImageNet is used as the pretraining dataset.  ", "questions": "## Questions:\n1. Line 125:  \"In our experiments, we find the lower the layer is, the better it performs, so we choose the first layer as default, such that k = 1.\".  Did you compare with pixel-level perturbations?  i.e. instead of perturbing the feature space, perturb the pixels (this would be equivalent to prior work ike PGD, FGSM etc.). How does PAPs method compare against these?\n2. In Table 1, 2 -- UAPEPGD seems to perform significantly better than PAPs for certain datasets (eg. CIFAR10), but on an average PAPs is better -- what might be the reason for this?  Please provide some intuition for this.  \n3. It is unclear why Sec 4.5 (tradeoff between robustness and clean accuracy) is needed.  This is a well-known result as the paper itself cites Zheng et al for example.  Since the setting of this paper is different, as mentioned by line 250-252, what is the new insight that is being conveyed through Sec 4.5 / Fig 7?\n\n## Suggestions about Formatting, Typos, Structure etc. (Note: these points do not affect my rating, but will be better to address)\n1. Please increase the font size for tables 1, 2, 3 and figure 7.\n2. Please also include a section on adversarial defenses, and (if there are) techniques developed for defending against trasferable / universal perturbations.  \n3. Line 114: \" To alleviate the lousy effect of the above negative factors\" -- the term \"lousy\" seems out of place here.  I would probably just rephrase it as \"To alleviate the negative effect of the above factors\".\n4. In Table 1, 2, 3, please mention that the % numbers are error rates -- this isn't mentioned anywhere in the paper. \n", "limitations": "Limitations and social impact are mentioned in Sec 6.\nThis section should also mention that finding from previous sections -- that UAP is a better attacker than PAPs on the pretraining domain -- as a limitation. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n1. Recently, pretrained models have been shown to be useful for many downstream tasks, and in many domains (such as vision, NLP, robotics, audio, ..).  Additionally, \"prompt\" learning has made pretrained models easy to apply in a zero-shot setting.  Due to these reasons, the idea of transferable adversarial perturbations (i.e. learned on the pretrained model, but also affecting unseen downstream tasks) is interesting and useful for the community.\n2. Sec 5 discussion on why UAP is better than PAP on the pretraning domain, is a good contribution.\n3. The finding from Fig 2 is interesting -- it shows that lower-level features of the classifier do not deviate much after finetuning on downstream tasks.  \n\n## Weaknesses:\n1. The proposed method is tested only on image classification tasks.  Often pretrained models are used for many different tasks (for eg. in NLP, pretrained language models can be used for text classification, text generation, machine translation, question answering, etc.  Recently, there has also been an effort to use pretrained vision-language models for many different downstream tasks.  This paper falls short of evaluating the adversarial perturbations on *different* tasks (not just different datasets for the same task).  See:\n    i. https://arxiv.org/abs/2204.14198\n    ii. https://arxiv.org/abs/2104.00743\n2. Eq 6 ($PAP_{UGS}$) method, uses information from the finetuning datasets -- this should be clearly mentioned in the paper.  It only becomes apparent after reading Eq 6.  \n\nOverall -- the paper does a good job at defining a new problem statement, providing a method, and showing empirical results on one task (image classification).  However, I find it lacking in terms of transferability to other tasks (see weakness 1).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kLwbE6vwBXZ", "reviewer": "Reviewer_2UeR", "summary": "This work proposes a new type of adversarial attack that aims at the pre-trained models, which could still fool the pre-trained model after fine-tuning. The proposed method is built upon the observation that the lower layers of the pre-trained model hardly change after fine-tuning. Specifically, they propose to generate a perturbation that can fool the feature of lower layers. Empirical results demonstrate a high attack success rate after fine-tuning.", "questions": "My concerns and questions are listed in the weakness part.", "limitations": "No social negative impact is found here.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n1. Figure 2 nicely shows the feature in the lower layers only change a little.\n2. The performance on multiple datasets is reported.\n\nWeaknesses\n1. For the motivation, I am wondering if this setting is realistic. In practice, it is way harder to get the pre-trained model than the weight of the model. Moreover, knowing the weight of the model can make the adversarial attack much more successful (since we can attack both low and deep layers). In this case, why do we still want to attack the pre-trained model?\n\n2. For evaluation, the authors only attack the standard pre-trained models. I think it is necessary for conducting experiments on the adversarial pre-trained models, which are obviously more robust than the standard pre-trained models.\n\n3. For evaluation, the fine-tuning setting is not described. Which fine-tuning is performed here, adversarial or standard fine-tuning? If it's adversarial fine-tuning, which adversarial fine-tuning method is employed here? If it's standard fine-tuning, what's the attack success rate for adversarial fine-tuned models.\n\n4. There is no detailed description in the caption of Table 1, Table 2, and Table 3. It is hard to learn the meaning of the percentage value in these Tables", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "foYP2q_ykLf", "reviewer": "Reviewer_qc7g", "summary": "This paper aims to propose a new attack method that can specifically improve the attack success rate on big pre-trained models. The newly proposed two techniques, Low-Level Layer Lifting Attack and Uniform Gaussian Sampling, have demonstrated notable performance improvement. ", "questions": "Please refer to my previous questions on the weakness part.", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n(1) The proposed methods sound reasonable. Damaging the low-level features of the pre-trained models intuitively should have the ability to attack the downstream tasks. \n\n(2) The experimental results have demonstrated the effectiveness of the proposed methods, outperforming the previous methods by a large margin. \n\n(3) The analysis of the proposed method is quite extensive, almost covering all the aspects I can think of. The trade-off between the clean accuracy and robustness is interesting.\n\n## Weaknesses\n\n(1) All the figures and tables are two small. I need to zoom in by 300% to clearly see the contents.\n\n(2) I am curious about the formulation of Eq4. Why don't we maximize ||f(x + δ) - f(x)||, but to maximize -f(x + δ)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fmXu7PcwjQC", "reviewer": "Reviewer_z1MX", "summary": "This paper tries to attack the pre-trained models (when handling the downstream tasks) by introducing a Low-Level Layer Lifting Attack (L4A) method, which mainly perturbation the neurons lying at the low-level layers of the pre-trained models.", "questions": "For the `Low-Level Layer Lifting Attack`, the authors chose the first and second layers of the pre-trained to attack since they found the layers laying at a higher level, the more their parameters change during fine-tuning. However, this does not mean that attacks the parameters at the higher level layers are useless. Hence, some additional experiments can be done. For example, attack all the parameters at different layers but with some hyperparameters (i.e., loss weights to balance each term). Also, I think there should be a hyperparameter in `Eq.5` between those two norm.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe method looks simple and effective. It greatly improved the attack success rate compared to the SOTA.\n\n\nWeaknesses:\n\n1. Figure 1 gives little information about this method, and I think more detailed information should be given in the caption. \n2. The author just uses three pre-trained models, two of which are similar (ResNet50 and ResNet101); how about the performance on CLIP or the other pre-trained models?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept