Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness
Authors: Yibo Xu
Dawei Zhou
Decheng Liu
Nannan Wang
Venue: ICML 2025 poster (2025)
Keywords: Adversarial attack
Adversarial robustness
Prompt-based defense
Primary area: deep_learning->robustness

Abstract: Deep neural networks are found to be vulnerable to adversarial perturbations. The prompt-based defense has been increasingly studied due to its high efficiency. However, existing prompt-based defenses mainly exploited mixed prompt patterns, where critical patterns closely related to object semantics lack sufficient focus. The phase and amplitude spectra have been proven to be highly related to specific semantic patterns and crucial for robustness. To this end, in this paper, we propose a Phase and Amplitude-aware Prompting (PAP) defense. Specifically, we construct phase-level and amplitude-level prompts for each class, and adjust weights for prompting according to the model's robust performance under these prompts during training. During testing, we select prompts for each image using its predicted label to obtain the prompted image, which is inputted to the model to get the final prediction. Experimental results demonstrate the effectiveness of our method.

Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness
Yibo Xu Dawei Zhou'
Decheng Liu'
Nannan Wang'
Abstract
Previous defense
Deep neural networks are found to be vulnerable to adversarial perturbations. The prompt-based defense has been increasingly studied due to its high efficiency. However, existing prompt-based defenses mainly exploited mixed prompt patterns, where critical patterns closely related to object semantics lack sufficient focus. The phase and amplitude spectra have been proven to be highly related to specific semantic patterns and crucial for robustness. To this end, in this paper, we pro- pose a Phase and Amplitude-aware Prompting (PAP) defense. Specifically, we construct phase- level and unplitude-level prompts for each class, and adjust weights for prompting according the model's robust performance under these prompts during training. During testing, we select prompts for each image using its predicted label to ob- tain the prompted image, which is inputted to the model to get the final prediction. Experimental re- sults demonstrate the effectiveness of our method.
Prompt with mixed patterms
Target
model
Rabbit
X
Tested exomple
Proposed
Amplitudetevel prompe
Amplitade spectrum
Target
Phase spectrum
Bird
example
Phase-level prompt
Different semaantic patterns
Hlgh-frequency
Low-Frequency
Phase
Advensrial Natural
Figure I. Differences between previous defenses and our defense. Previous method mixed patterns like pixel frequency do- mains for prompting. However, they do not explicitly focus on specifie semantic patterns. The phase and amplitude spectra can flect structures and textures specifically. Our method utilizes these specific patterns for prompting, further improving the robustness.
1. Introduction
2022: Zhou et al., 2023). However, existing prompt-based defenses mainly focus on mixed patterns, such as pixel and frequency domains (see Figure 1). These patterns cannot explicitly reflect specific patterns like structures and tex- tures. To this end, we seek to disentangle the mixed patterns, and construct prompts for stabilizing model predictions by utilizing patterns closely related to the object semantics.
Deep Neural Networks (DNNs) have been found to be vul- nerable to adversarial noises (Szegedy et al., 2014; Xiao al., 2018; Yang al., 2023). This vulnerability has posed significant threat many deep learning applications (Jaiswal et al., 2022; Mi al., 2023; Shukla et al., 2024), promoting the development of defenses (Madry al., 2018; Zhou et al., 2022: Zhao et al., 2024; Xia et al., 2024).
The amplitude and phase spectra of the data have been proven to be able to reflect the specific semantic patterns. Previous studies indicated the amplitude spectrum holds texture patterns (Randen & Husoy, 1999; Sidhu & Raa- hemifar, 2005), while the phase spectrum reflects structural patterns (Kovesi, 2000; Zhang al., 2011). Besides, cogni- tive sciences reveal that people tend recognize objects by utilizing the phase spectrum (Freeman & Simoncelli, 201 Gladilin & Fils, 2015), which can also help improve the model's generalization ability (Chen et al., 2021). Also, the amplitude spectrum has been proven to be easily manip- ulated by noises and thus further processes are needed to mitigate this problem for robustness (Chen al., 2021). To
Recently, prompt-based defenses have been increasingly investigated (Huang et al., 2023; Chen al., 2023; Zhou al., 2024). It of interest since it does retrain target models like adversarial training does (Wu et al., 2020; Wei al., 2023; Singh al., 2024), and does not perform major modifications on data as in denoising methods (Nie et al.,
Xidian University, Xi'an an, Shaanxi, China. Correspon- dence to: Nannan Wang mamwan@@xxdian.dduu <dwzhou.xidin xidiannuxxiii@eeeee giteooouiddiin@mmal..o.... .om..
Dawei Zhou
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Phase and amplitude-wware Prompting for Enhancing Adversarial Robustness
this end, constructing prompts using amplituee and phase spectra is expected to provide positive effects for prompt- based defenses (see Figure 1).
directly select the prompts for images based on their predicted labels for testing.
We evaluate the effectiveness of our method for both naturally and adversarially pre-trained models against general attacks and adaptive attacks. Experimental results reveal that our method outperforms state-of-the- art methods and achieves superior transferability.
Motivated by the above studies, we propose Phase and Amplitude-aware Prompting (PAP) defense mechanism, which constructs phase and amplitude-leve prompts to sta- bilize the model's predictions during testing. We learn a phase-level prompt and an amplitude-Ievel prompt for each class, since it can help learn more precise semantic patterns while reducing computational costs compared with learning prompts for each instance. Naturally, a question arises here: Do amplitude-led the same effect on the model robustness? To answer it, we utilize phase and amplitude spectra of natural examples to replace the corresponding spectra of adversarial examples respectively for testing as in Table 1. It shows phase and am- plitude spectra have different effects on model's predictions. Furthermore, we construct phase-level and amplitude-leve prompts, training them under different prompting weights. Table 2 shows different weights lead to different robustness, and thus we need to adjust their weights appropriately.
2. Related Work
prompts and phase-level prompts have
2.1. Adversarial Attacks
Adversarial attacks craft malicious noises to mislead target models. White-box attacks like Projected Gradient Descent (PGD) attack (Madry et al., 2018), Autokttack (AA) (Croce & Hein, 2020), Carlini& Wagner (C&W) (Carlini & Wagner, 2017) and Decoupling Direction and Norm (DDN) (Rony et al., 2019) craft noises through accessing and utilizing models' intrinsic information like structures and parameters. For black-box attacks like transfer-based attacks and query- based attacks (Andriushchenko et al., 2020), attackers have access to the models' internal information, and thus perform attacks only by interacting with the model's inputs and corresponding outputs.
Based these analyses, we propose weighting method for our prompts. Since different weights for prompting lead to different robust performances, we adjust their weights based on their influences on robustness during training. We adjust the weight for amplitude-leve prompts by the ratio of ac- curacy under adversarial training examples with amplitude- level prompts to that with phase-level prompts, since the ratio can reflect the relative importance of amplitude-level prompts compared to phase-level prompts.
2.2. Adversarial Defenses
Adversarial training methods (ATs) (Madry et al., 2018;
Zhang et al., 2019; Wang et al., 2019) aim at augmenting training examples through adversarial noises for training. However, ATs require modifying parameters of models and crafting noises for training, consuming significant resources. In addition, denoising methods (Jin et al., 2019; Zhou et al., 2023) purify images before feeding them into target models. It introduces an additional module for substantially modi- fying data to remove noises, thereby also consuming great computational resources.
During testing, we select prompts for each image according to the model's predicted label for it. Previous method (Chen et al., 2023) traverses all the prompts for different classes for testing, causing great time consumptions especially on datasets with many classes. To alleviate this problem, we directly select prompts for tested images according to their predicted labels. To further reduce the negative effect of mis- matches between images and selected prompts, we design loss that helps images with prompts not coming from their ground-trutt -truhh labels to still be correctly classified. Our code is available at https: / giithub... com/ www//yeeoox..ccccccoccocooooooo yeebox// PPP.
To alleviate this problem, prompt-based defenses has at- tracted more and more interests due to its efficiency (Huang et al., 2023; Chen al., 2023; Zhou al., 2024). C-AVP (Chen et al., 2023) trains pixel-level prompts for each class, and traverses all the prompts for testing. However, it re- quires high computation costs on datasets with numerous classes. Frequency Prompting (Freq) (Huang et al., 2023) aims at mitigating the vulnerability of models in the high- frequency domain by masked prompting strategy. How- ever, it does not explicitly focus on specific semantic pat- terns, where the semantic pattern which C-AVP focused on is also mixed (i.e., the pixel domain). Differently, we focus on specific textures and structures by prompting on the amplitude and phase spectra. Also, our method does not traverse all the prompts for testing, achieving superior performances efficiently through selecting prompts using
Our contributions can be summarized as follows:
Considering the amplitude and phase spectra are closely related to specific semantic patterns and cru- cial for robustness, we seek design phase-level and amplitude-Ievel prompts to provide positive gains for prompt-based defenses.
We propose a Phase and Amplitude-awar Prompt- ing (PAP) defense. Specifically, we propose a weight- ing method for prompts based on their impacts on the model's robust performances for training, and propose
2

---

Phase and amplitude-wwaee Prompting for Enhancing Adversarial Robustness
Training
Inference
F
F&
nat
Test example
Natural example
Wt
Target model he
P
Pty
F+
ha
Ypred Select
x W
Ladv
Adversarial example
F$
sim
W
Fg
Po Ppred
Pyyprod
Ptyr
X We
Target model hg
mis
Adversarial
Poy
Target model hg Bear
example
F
Figure 2. The framework of method. First, We construct phase-level and amplitude-level prompts, and adjust the weights of amplitude- level prompts according to their influences on robustness for training. Then, we select prompts from predicted labels to get prompted images using the finally adjusted weights for testing.
predicted labels.
et al., 2001; Ren et al., 2015). Thus, we seek to further dis- entangle these semantic patterns for enhancing robustness.
3. Methodology
The phase and amplitude spectra have been proven to be able reflect specific semantic patterns. Through Fourier transform, image signals in the pixel domain can be con- verted into the frequency domain, which can be further decoupled into phase and amplitude spectra. The phase spectrum can reflect structures (Kovesi, 2000; Zhang et al., 2011), while the amplitude spectrum carries textures (Ran- den & Husoy, 1999; Sidhu & Raahemifar, 2005). Cogni- tive sciences indicate people tend recognize objects by leveraging structures from the phase spectrum (Freeman & Simoncelli, 2011; Gladilin & Eils, 2015), which has been proven to be able to help DNNs improve their generaliza- tion performances (Chen et al., 2021). Also, the amplitude spectrum has been analyzed to easily manipulated by noises, indicating the necessity mitigate this problem for robustness (Chen et al., 2021). To this end, constructing phase-level and amplitude- mixed patterns is considered to be beneficial for improving prompt- prompt-based based defenses (see Figure 1).
3.1. Preliminary
In this paper, focus classification tasks under adver- sarial settings. Given model he with parameters and natural data (x. y). the adversarial example ă is crafted for misleading hg. Since we focus on images, we utilize Discrete Fourier Transform (DFT) and its inverse version (IDFT), denoted as F()) and (-.),, respectively. The phase and amplitude spectra are derived as Φ2 (a) and & = (x). We use dh and &. to denote phase and amplitude spectra of a natural image æ, while da and & de- note the corresponding spectra of ž. In addition, the process to recover an image from its phase and amplitude spectra expressed as x = F Ex). Our goal is to design prompts to assist bg in making accurate predictions without the need of the model retraining.
prompts to disentangle the
3.2. Motivation
DNNs can be casily fooled by adversarial noises. Defenses like adversarial training and denoising methods all improve robustness with high computational cost, promoting the development of prompt-based defenses due to the efficiency. However, existing prompt-based defenses focus mixed patterns like pixel or frequency information, which cannot capture specific patterns like structures and textures (Ying
3.3. Defense
Based the above analyses, we introduce the designed Phase and Amplitude-awai Prompting (PAP) defense. We first construct prompts and adjust the weights of amplitude- level prompts based on their influences on robustness. Dur- ing testing, we select prompts from predicted labels for
3

---

Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness
prompting. The framework is shown in Figure 2.
3.3.1. 1. PROMPT CONSTRUCTION AND TRAINING
(Pty
We
We firstly construct and train a phase-level prompt and an amplitude-level prompt for each class, since it can help learn precise natural semantic patterns for each class while reduc- ing computational costs compared with learning prompts for each instance. We randomly sample a natural example a from class y, and obtain its phase spectrum Φ₂ (x) and amplitude spectrum & (z) the prompt initialization for class y. The prompt initialization for other classes is performed following the above operation. Then, the initial- ized phase-level and amplitude-level prompts are denoted as {pp: Jiz0 and (pc.jio. 14-1 where is the number of classes. The prompted image xº for obtained follows:
F
Target
Update Wi
model
F
hg
Adversarial examples
[Poy, 12m1
Figure 3. The weighting method for amplitude-love prompts. We use amplitude-level and phasc-level prompts respectively for prompting, and adjust weights by the ratio of accuracy of images with amplitude- prompts that with phase-level prompts.
(1) where Po, and Pe, denote the phase-level prompt and the amplitude-level prompt corresponding to the ground-truth label y of r. We perform prompting for the adversarial example 전 following the same way as Equation 1. Then, the designed training losses are introduced as follows:
I F.Fle ( =
+ Poy: Ea + ),,),
Data-prompt Mismatching Loss. Note that we learn phase-level prompt and an amplitude-level prompt for each class. Also, we select prompts according to predicted labels during testing. Therefore, there exist mismatches between test images and selected prompts when testing. Previous studies (Chen et al., 2023) indicate we can get prompted images using prompts from classes different from ground- truth labels, and enforce their outputs on ground-trus truth labels to be larger than those on other labels, so that these images can still be correctly classified to some extent. To this end, we construct a data- aata-prompt mismatching loss as:
Classification Loss. To enforce our prompts to stabilize the model's predictions, promote our prompts learn to help correct wrong predictions of models, and thus exploit the prompted examples to construct the classification loss:
1 Σ
N
Cado
Iyllog(he he (2))),
Liis
Σ
Amax Wh (Z)) 11 (Z) ( (5)
N
where N is the number of examples, and x.. denotes the prompted image of the adversarial example ā; using the prompts from its ground-truth label y/- Then, the classifica- tion loss for the natural prompted data presented as:
where 2. is the prompted adversarial example using prompts from s/, which is a randomly selected label differ- ent from its ground-truth label yı. 1 (-) and h (-) denote outputs on of and ys, and r is a threshold.
N
Lnat
(3)
3.3.2. WEIGHTING METHOD
The designed prompts with the corresponding training pro- cedure focus on helping the model make predictions accu- rately during testing. However, it is natural for to ques- tion whether the amplitude-Ievel and phase-level prompts have the same influence on the robustness. To this end, we conduct several experimental analyses to answer it.
where xp denotes the prompted image of the natural example I; using the prompts from its ground-truth label yi.
Reconstruction Loss. The constructed prompt could mod- ify the phase and amplitude spectra during prompting. To ensure these modifications do not severely disrupt the orig- inal semantic patterns, we design reconstruction loss be- tween the prompted adversarial images and natural images
We firstly replace the amplitude and phase spectra of adver- sarial examples with the corresponding spectrum natural examples respectively. As shown in Table for different models, the natural amplitude spectrum and natural phase spectrum contribute differently to the model's robust perfor- mances. It indicates that the amplitude and phase spectra have different influences on the model's 's robustness.
N
H
W el Jine
Lrim NXHXW
(4)
where n.. and my.k denote the pixel value of a and the pixel value of i; in the j-th row and k-th colum respectively, and H. W denote the height and weight of the image.
Due to their different influences on robustness, we may need to assign different weights for phase-level prompts and

---

Phase and Amplitude-wware Prompting for Enhancing Adversarial Robustness
Algorithm Phase and Amplitude-aware Prompting (PAP).
Table The impact of different spectra on robustness. Adv All denotes normal noises. Nat. Pha//Amp. indicates we replace phase/amplitude spectra with corresponding natural spectra.
The target model hss training dataset D. batch size n. the number of batches M. epoch number Z', perturbation budget e, the initialized phase-level prompts Nizo and amplitude-Ievel prompts {pe.)-md-
Input:
Adv. All
Nat. Pha.
Nat. Amp.
2: for = 1 to 1' do
NAT
0.00
47.81
13.41
3: for m = 1 to M do
AT
46.82
69.00
70.10
4:
Read mini-hatch B Ne= from training set D;
5: Craft corresponding adversarial samples B = ( Hix at the given perturbation budget
Table 2. Robust accuracy (percentage) under different weights. a and are weights of phase-level and amplitude- prompts.
6:
Calculate Lau by Equation 8 optimize (ps, Jimo and
{pe, Jizo end for
7:
a
8
None
NAT
PGD
None
AT
PGD
9: update ue, via Equation 6;
8:
if mod 0 then
1
0.01
88.82
34.44
84.31
47.69
10: end if
0.01
1
91.11
5.64
73.84
50.87
11:
end for
amplitude-level prompts to further improve the robustness. To show this, we assign different weights for them to obtain the prompted images for training and testing. As shown in Table 2, it is clear that different weight assignments result in different robust performances. Therefore, we need to design a strategy which can appropriately adjust their weights.
3.3.3. OVERALL DEFENSE PROCEDURE
To improve the overall effectiveness of our combined de- fense, we incorporate the the weighting strategy into the training process. The overall loss function is denoted as:
Lall = Lado + A Lmnt + AeC + Aghmiss where Å1. X2, À are syper-parameters.
(8)
Based on it, we enforce prompts to assign weights for them- selves according to their influences on the robustness. The robust accuracy after prompting can explicitly reflect the influence of these prompts on robustness, which has been proven to be suitable for measuring the importance of them and adjusting their weights (Wang et al., 2019; Wei et al., 2023). Therefore, we use the robust accuracy under these prompts to adjust their weights. Specifically, during training, we obtain robust accuracies in training data using amplitude- level and phase-level prompts for prompting respectively. Then, the weights for amplitude-level prompts are adjusted by the ratio of accuracy under amplitude-level prompts to accuracy under phase-level prompts, since it can reflect the relative importance of amplitude- prompts compared with phase-level prompts for robustness. The weight strat- egy is specified as:
The overall defense procedure is presented in Algorithm 1. Specifically, during training, for each mini-batch B, we craft adversarial examples B. Then, to calculate Lalt using Equation 8, and further optimize phase-level prompts (P&: /i=o and amplitude-level prompts Ji- The weight for amplitude-Ievel prompts is ad- justed by Equation 6. Through iteratively optimizing the prompts and adjusting the weights, the prompts are expected to provide superior robustness gains.
forward-pass B and B
ye-1
3.3.4. PROMPT SELECTION FOR TESTING
After acquiring our prompts, need to explore effective prompt selection method during testing. Previous methods (Chen et al., 2023) traverse all the prompts from all the classes for testing on naturally pre-trained models, which sets the label with the largest output among all the prompting cases as the final prediction (see Appendix B). However, it can easily cause high computational costs for testing on large datasets with numerous classes. To address it, we promote the test image to choose prompts corresponding to its predicted label directly. Incorporated with the learned weight, we obtain the prompted image for testing as:
EN I(f(E = v1) Li
Wi = 11-1
IN IIf(" = yi)
(6)
where T/e F-1( + wt wt-1 and Z. F (0s. +Po., (x.). and yi is the ground-truu -tuth label of the adversarial example 2, II() denotes the indicator function, and wi is the weight during the t-th epoch. The Equation then incorporated with the designed weight as:
#test (Drun + Paryreaa Exre + w" Pesprea
x = F (chnr + Poy: Exr + W.PE,),
(7)
where Peryrad and Psnyrod are selected prompts from the predicted label ypred- Since this strategy may result in mis- matches between images and selected prompts, we introduce a data- aata-prompt mismatching loss to alleviate its negative ef- fects on robustness, which can be seen in Section 3. 1.
where we use Equation 7 for training. The finally learned weight w" is utilized for testing, and is shown in Table 5. The weighting strategy is illustrated in Figure 3.
5

---

Phase and Amplitude-wware Prompting for Enhancing Adversarial Robustness
Table Robust accuracy (pereentage) of defenses against adversarial attacks CIFAR-10 and Tmny-ImageNet. The target models are R8 and WRN28- 10. We present the most successful defense results with bold.
CIFAR-I0 (ResNet18)
Tiny-ImageNet (WRN28-10)
Defense
None
AA
C&W
DDN
None
AA
C&W
DDN
None
94.83:0.055
0.00+00.00
0.00++0.00
0.00
6..62
0.00=00.00
0.00++0.00
0.02+00.00
+Freq
94.50++021
0.44+0.08
11.57++0.15
4.60++0.07
60.43+0.17
2.56+0.03
14.82++0.20
10.51+0022
+C-AVP
92.6710.51
0.6110.11
1.93-10007
1.0510.17
66.52:10.02
0.39=0.00
5.83:10.07
4.19:0.25
+PAP(Ours)
87.12110.21
37.34-10.11
80.27110.28
66.22.10.21
57.3010.15
5.33-0.07
42.14:10.36
33.2710041
AT
84.22-10.21
44.94
0.8410.18
2.9710.34
5I.3910.17
18.2910..2
0.1910..22
TI4880027
+Freq
78.26+0.11
51.50+0.31
35.67+0.35
35.12=0.11
44.84+t0.28
22.93+00.66
19.25+0.41
21.76+00.00
+C-AVP
84.28+0.24
45.79-+0.3
11.00=t0.14
10.35=0.24
51.16+0.07
19.82+0.21
12.42+0.19
17.45+0.22
+PAP(Ours)
84.34+0.12
52.31+0.119
66.66+0.18
60.29=0.07
51.40=
23,44+0.24
35.76++0.34
04.98+0.16
TRADES
813591022
48,881022
0.14I60.0
3,0$10.09
48.9816.00
178110.07
0.1510.000
1I4.09000009
+Freq
75.62:10.22
52.87-1..24
30.98:10.15
30.7410.05
41.8010.111
21.64.10.3
15.47-10.07
21.36:10.20
+C-AVP
81.58:10.17
49.4411022
4.47
1.97
48.59:10.19
19.15
9.18-0.177
18.57
+PAP(Ours)
01.62:1.114
54.36:10.24
63.89:0.27
57.58=0.20
48.03++024
22.73+0.34
30.43:1014
0.2
MART
00.31=0.24
46,95-10.24
0.75+0000
3.85
44.29-10.04
19.1
0.34
15.31
+Freq
74.1410.18
52.60+0.22
34.40:L0.16
32.79=0.19
37.76+10.28
22.80=0.27
15.4611..24
21.75=:0.24
+C-AVP 80.29+0.25
47.25+0.27
3.51+00.55
6.11+0008
43.86+00.27
20.66++0.20
10.78+0.33
20.09+-0.27
+PAP(Ours)
79.49+0.20
53,79++0.11
60,36++022
56.66-0.30
43.71+0023
23.74++.27
30.39+00.09
32.18+0008
Attack Settings. We use various adversarial attacks across two norms for evaluations. Specifically, we utilize Loe- norm AA (Croce & Hein, 2020), L2-norm C&W (Carlini & Wagner, 2017) and L2-norm DDN (Rony al., 2019). The iteration number of L2-norm DDN is set as 20, while that of Ly-norm C&W is 50. The perturbation budget for L -orm AA is set as 8/255.
Table Robust accuracy (percentage) of defenses against adver- sarial attacks CIFAR- 10 using the prompt selection method of C-AVP. The target model is ResNet18.
Defense
None
AA
C&W
DDN
NAT
94.83
0.00
0.00
0.00
+Freq
56.80
7.54
49.22
30.25
+C-AVP
52.17
32.26
46.78
39.51
+PAP(Ours)
86.59
38.33
83.88
70.24
Defense Settings. We use prompt-based defenses C-AVP (Chen et al., 2023) and Freq (Huang et al., 2023) as base- lines, where they are designed only for defending on natu- rally pre-trained models. We use the natural training (NAT), AT (Madry et al., 2018). TRADES (Zhang et al., 2019) and MART (Wang et al., 2019) to obtain pre-trained models. We use PGD with perturbation budget 8/255, perturb step and step size 2/255 for training. We train prompts by SGD (Andrew & Gao, 2007) for 100 epochs, where the initial learning rate is 0.1 and is divided by 10 at the 75-th epoch. The batch size is 512 for CIFAR- 10, and 256 for Tiny-ImageNet. We set A = 3, 400400, X3- 4 for natu- rally pre-trained models, and λ = 1, X2 = 5000, À3 = 4 for adversarially pre-trained models. The threshold 7 is set as 0.1, and we adjust the weights amplitude-level prompts every epochs. We omit deviations several tables due to their small values (50.60%) More details of settings can be found in Appendix C.
Table 5. The learned weights for the amplitude-level prompts. We show the results of ResNet18 and WRN28-10.
Model
Dataset
None
AT
TRADES MART
ResNet18
CIFAR-10
0.3054
0,2572
0.3258
WRN28-10 10
Ting-lmageNet
0.2702
0.3022
0.2848
4. Experiments
4.1. Experimental Settings
Datasets and Models. We use two popular benchmark datasets CIFAR-10 (Krizhevsky et al., 2009) and Tiny- ImageNet (Le & Yang, 2015) for defense evaluations. CIFAR-10 has 10 classes with 50,000 training images and 10,000 testing images, and Tiny-ImageNet has 200 classes with 100,000 training images, 10,000 validation images and 10,000 testing images. All the images are normal- ized into J0, 1]. We use ResNet18 (He et al., 2016) and WideResNer28-10 (WRN28-10 10) (Zagoruyko, 2016) as tar- get models, and use WRN28-10, VGG19 (Simonyan & Zisserman, 2014) and popular Swin Transformer (Liu et al., 2021) for evaluating the transferability of defenses across different models.
4.2. Defending against General Attacks
Defending against White-box Attacks. We apply vari- ous attacks to evaluate the robustness of our method and baselines. The average accuracies with the deviations are presented in Table 3.
Figure 4 shows our method can preserve complete semantic
6

---

Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness
Table Robust accuracy (percentage) of defenses on different models. All the prompt-hased defenses trained R8sNet18, and then applied to the VGG19 and WRN28- respectively. We present the most successful defense results with bold.
CIFAR-10
Tiny-ImageNet
Model
Defense
None
AA
C&W
DDN
None
AA
C&W
DDN
NAT
93.13
0.00
0.00
0.00
59.40
0.00
0.00
0.03
+Freq
90.77
21.08
11.07
47.79
4.27
16.58
12.55
+C-AVP
21.60
10.78
15.58
12.32
50.95
3.00
17.97
13.61
+PAP(OurS)
87.65
34.28
81.33
65.65
49.30
7.39
38.66
31.26
VGG19
AT
80.12
42.61
0.38
3.42
38.99
10.73
0.13
5.40
+Freq
73.78
49.04
34.08
33.10
30.11
15.69
16.23
16.93
+C-AVP
79.99
43.60
10.67
10.41
38.86
14.20
16.85
16.60
+PAP(Ours)
79.53
49,43
56.26
51.22
38.37
16.55
29.98
27.76
NAT
95.42
0.00
0.00
0.00
66.62
0,00
0.00
0.02
+Freq
94.53
0.98
18.29
7.95
54.47
3.67
18.29
13.78
+C-AVP
10.24
9.95
10.32
10.14
61.83
2.07
19.11
13.49
+PAP(OurS)
87.83
41.81
81.88
69.19
54,93
6.88
40,90
34.05
WRN28-10
AT
87.85
49.75
7.16
4.62
51.50
T8.27
0..5
1T.48
+Freq
82.50
54.40
34.79
34.54
44.29
23.32
19.88
22.18
+C-AVP
87.74
50.50
12.01
12.93
51.18
20.03
15.31
19.31
+PAP(Ours)
87.26
54.99
70.29
63.62
51.27
23.71
36.27
35.41
Table 7. Robust accuracy (percentage) of defenses against adaptive attacks CIFAR-10. The target model ResNet18.
Input examples
Defense
None
AdaA20
AdaA40
Freg
NAT+Freq
94.35
0.68
0.88
NAT+C-AVP
77.74
0.01
0.01
NAT+PAP(OurS)
90.44
27.95
17.64
C-AVP
AT+Freq
78.30
"32.44
33.222
AT+C-AVP
83.08
45.32
45.08
PAP(OurS)
AT+PAP(OUrs)
84.70
47.94
47.04
Figure 4. Visualizations of prompted images for input examples. each pair images, the left part denotes prompted image, while the right part denotes the difference heatmap compared to the original input (i.e,, adversarial) example.
based attacks using VGG19 as the surrogate model and query-baser attack Square (Andriushchenko et al., 2020) for evaluations. Table 12 Appendix D shows our method achieves superior performances, verifying the practicality of defense in real scenarios.
patterns after prompting. Quantitative analyses in Table 3 show that our method improves the robustness by a large margin on various attacks compared with existing defenses. On Attokt by about 37% CIFAR-10 and 4% Tiny-ImageNe for naturally pre-trained models, and achieves better robustness on adversarially pre-trained models from both datasets. Al- though Frequency Prompting improves robustness against AA for adversarially pre-trained models to some degrees, decreases natural accuracy by about 7% on these models. On C&W and DDN, our method provides great positive effects for robustness. In addition, although our method sacrifies natural accuracy on naturally pre-trained models to some extent, it greatly improves robustness against all of these attacks (c.g., 80.27% and 42. 14% against C& W on CIFAR- 10 and Tiny- ImageNet).
Defenses on the Prompt Selection Method of C-AVP. To further verify the stability of our method, we use the prompt selection strategy of C-AVP on the naturally pre-trained ResNet18 on CIFAR-10 for evaluations. Table 4 shows al- though previous methods achieve some improvements on robustness. they reduce natural accuracy by large mar- gin. In comparison, our method can still protect models more effectively without losing natural accuracy too much, achieving more stable performances.
PAP helps increase the robust accuracy
4.3. Defense Transferability
To evaluate the transferability across different models, applied our method trained on ResNet1 8 to other target models, i.e., WRN28-10, VGG19 and Swin Transformer. Table and Table 13 (see Appendix E) show our PAP can effecttvelyy help defend against various attacks across both convolutional neural networks and vision transformers. It in-
Defending against Black-box Attacks. We apply transfer-

---

Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness
Table The impact of different losses on CIFAR-10. We report
Table 10. The effectiveness of learning prompts for each class on
average robust accuracies due to space limitations.
CIFAR-I0 -O using ResNet18 compared with universal prompts.
Losses
NAT
AT
Defense
None
AA
Defense
None
AA
Laat
Lisi
Limis
None
Avg
None
Avg
NAT
94.83
0.00
AT
84.22
44.94
48.27
41.35
84.32
59.77
+Universal
87.54
31.81
+Universal
84.56
51.92
X
83.64
61,56
83.20
54.19
+PAP(Ours)
87.12
37.34
+PAP
84.34
52.31
10.01
10.43
84.34
59.48
87.12
61.27
84.34
59.57
Table 11. The effectiveness of defenses with Gaussian Blur on CIFAR-10. The target model is ResNet18.
Table The effectiveness the weight for prompting. We report average robust accuracies on both datasets.
Defense
None
AA
C&W
DDN
NAT
70.57
25.92
56.79
40.10
CIFAR-I0 10
Tiny- Mny-lmmeeNet
weight
+Freq
70.50
26.51
57.94
40.88
None
Avg
None
Avg
+C-AVP
65.83
25.75
53.01
37.04
75.92
57.27
41.59
28.70
+PAP(Ours)
67.83
46.57
65.38
58.85
84.34
59.57
51.40
31.39
AT
78.55
55,51
57.78
54.77
+Freq
72.72
55.94
57.57
55.61
+C-AVP
78.09
55.68
59.16
55.95
dicates that we can train our prompts only once and directly apply them to other models for defenses effectively.
+PAP(Ours)
77.01
57.41
64.57
61.68
4.4. Defending against Adaptive Attacks
Comparison with Universal Prompts. To verify the superi- ority of learning prompts for each class, we train a universal phase-level prompt and a universal amplitude-Iewe prompt for comparisons. For fairnsss, the data-prompt mismatching loss removed on universal prompts, and other settings for universal prompts the same as those from method. Table 10 shows performances under universal prompts are worse than those of ours, indicating that our method helps enhance the robustness.
Since we achieve defenses by prompting, the prompts could be leaked attackers for performing adaptive attacks (AdaA). In this case, attackers focus on crafting adversarial noises for misleading predictions after prompting as:
ma (F (Fo(zzt)) ((())) v), (10)
where tee denotes the cross-entropy loss and å is the per- turbation. For fairness, we retrain our PAP on this attack and apply the same adaptive attack strategy on baselines using their own prompts to retrain them. Then, the retrained defenses are evaluated on adaptive attacks. The iteration number of attack for training 10, while that for testing is 20 and 40. Table shows our method achieves better robust accuracy, verifying the effectiveness of our method.
Effectiveness When Blurring the Edges. Some attacks like DDN tend to disrupt edges of objectives. Therefore, it's natural to question whether robustness gains from our PAP come from edge blurring. To this end, apply Gaussian Blur on the test image for evaluations. As shown in Table 11, when blurring edges, our method can still achieve superior defenses, indicating the effectiveness of PAP does not come from edge blurring.
4.5. Ablation Studies
Loss Functions. We explore impacts of losses with differ- ent hyper-parameters. Table 8 shows removing any of these losses will damage performances, such the extremely accuracy drop when removing Crat or Lmis on NAT. Also, Appendix F shows natural and robust accuracies vary dif- ferently in various appee-prreettrr settings for these losses. As whole, the saprr-parameters we set achieve superior performances in both natural and robust accuracies.
5. Limitation
Despite the advances in adversarial defenses, method still has several limitations. First, our method sacrifices some natural accuracy when prompting on naturally pre- trained models. We will address it in the future such as using Contrastive Learning, since is a useful method for mitigating the trade-off problem between natural and robust accuracies (Kim al., 2020; Jiang et al., 2020; Xu et al., 2024). Second, we do not perform evaluations ImageNet due to the limited computational resources. However, we conduct experiments on Tiny-ImageNet which has been widely used. Liny-ImaeNNet-22 is larger and has larger resolution than CIFAR-10, with more numbers of classes than those of ImageNet- 100. Results show that our method
The Weighting Strategy. To verify the effectiveness of the weighting strategy, we remove it for evaluations. As shown in Table 9, the performances drop lot when removing the weighting strategy. Therefore, this strategy for dealing with different effects of phase-level and amplitude- prompts on robustness is necessary and rational.
8

---

Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness
achieves superior performances, leading to believe that method can also work well on ImageNet. We leave them to the future work.
attack via random search. In European conference on computer vision, pp- 484-501. Springer, 2020.
Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In 2017 iece symposium on security and privacy (sp), pp. 39-57. leee, 2017.
6. Conclusion
In this paper, we focus on specific semantic patterns for improving prompt-based defenses. has been proven that phase and amplitude spectra reflect structures and textures, and both of them need to be manipulated for robustness. Therefore, we construct prompts using these spectra, and propose a Phase and Amplitudeaaware Prompting (PAP) de- fense, which learns a phase-level prompt and an amplitude- level prompt for each class. Considering different influences of phase-level and amplitude-Ievel prompts for robustness, we design weighting method for them according to the ro- bustness under these prompts. To perform testing efficiently, we select prompts according to predicted labels, and design data-pree mata-prompt mismatching loss to mitigate the negative effects of mismatches between images and their selected prompts. Experimental results demonstrate our method helps defend against general attacks and adaptive attacks, achieving superior transferability. Overall, our defense ex- plores specific semantic patterns to improve performances of prompt-based defenses.
Chen, A., Lorenz, P., Yao, Y., Chen, P.-Y,, and Liu, S. Vi- sual prompting for adversarial robustness. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Chen, G., Peng, P., Ma, L.,, Li, J., Du, L.,, and Tian, Y. Amplitude-phane recombination: Rethinking robustness of convolutional neural networks in frequency domain. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 458 467, 2021.
Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020.
Elsayed, G. F., Goodfellow, I,, and Sohl-Diche Sohl-Dickstein, J. Adver- sarial reprogramming of neural networks. arXiv preprint arXiv: 10w.:180611466 11146, 2018
Acknowledgements
Freeman, J. and Simoncelli, E. P. Metamers of the ventral stream. Nature neuroscience, 14(9): 195-1121,,, 2011.
This work was supported part by the National Natu- ral Science Foundation of China under Grants U22A2096, 62036007 and 62306227, in part by Scientific and Techno- logical Innovation Teams in Shaanxi Province under grant 2025RS-CXTD-011 in part by the Shaanxi Province Core Technology Research and Development Project under grant 2024QY2-GJHX-11 in part by the Fundamental Research Funds for the Central Universities under Grant QTZX23042, in part by the Innovation Fund of Xidian University under Grant YJSJ25007.
Gladilin, E. and Eils, R. On the role of spatial phase and phase correlation in vision, illusion, and cognition. Fron- tiers in Computational Neuroscience, 9:45, 2015.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Huang, Q., Dong, X., Chen, D., Chen, Y., Yuan, L.., Hua, G., Zhang, W., and Yu, N. Improving adversarial robustness of masked autoencoders via test-time frequency-domain prompting. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 1600-1610, 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Jaiswal, S., Duggirala, K., Dash, A., and Mukherjee, A.
Two-face: Adversarial audit of commercial face recogni- tion systems. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pp. 381-392, 2022.
References
Andrew, G. and Gao, Scalable training of 1 1-regularized log-linear models. In Proceedings of the 24th interna- tional conference on Machine learning, pp. 33-40, 2007.
Jiang, Z., Chen, T., Chen, T., and Wang, Z. Robust pre- training by adversarial contrastive learning. Advances in neural information processing systems, 33:161999162100, 2020.
Andriushchenko, M., Croce, F., Flammarion, N., and Hein, M. Square attack: a query-efficient black-box adversarial

---

Phase and Amplitude-wware Prompting for Enhancing Adversarial Robustness
Jin, G., Shen, S., Zhang, D., Dai, F., and Zhang, Y. Ape-gan: Adversarial perturbation elimination with gan. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2842-3846. IEEE, 2019.
Rony, J., Hafemann, L. G., Oliveira, L. S., Ayed, I. B., Sabourin, R., and Granger, E. Decoupling direction and norm for efficient gradient-based 12 adversarial attacks and defenses. In Proceedings of the IEEE/CVE confer- ence on computer vision and pattern recognition, pp. 4322-4330, 2019.
Kim, M., Tack, J., and Hwang, S. J. Adversarial self- supervised contrastive learning. Advances in neural in- formation processing systems, 33:2993-2299, 2020.
Shukla, S., Gupta, A. K., and Gupta, P. Exploring the feasibility adversarial attacks on medical image seg- mentation. Multimedia Tools and Applications, 83(4): 11745-11768, 2024.
Kovesi, P. Phase congruency: A low-level image invariant. Psychological research, 64(2):13-148,, 2000.
Sidhu, S. and Raahemifar, K. Texture classification us- ing wavelet transform and support vector machines. In Canadian Conference Electrical and Computer Engi-
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
neering, 2005., pp. 941-944, IEEE, 2005.
Langley, P. Crafting papers machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Singh, N. D., Croce, F., and Hein, M. Revisiting adver- sarial training for imagenet: Architectures, training and generalization across threat models. Advances in Neural Information Processing Systems, 36, 2024.
Le, Y. and Yang. X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Liu, A., Tang, S., Liang, S., Gong, R., Wu, B., Liu, X., and Tao, D. Exploring the relationship between architec- tural design and adversarially robust generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4096-4107, 2023.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, Intriguing proper- ties of neural networks. In International Conference on Learning Representations, 2014.
Tsai, Y-Y., Chen, P.-Y,, and Ho, T.-Y. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In International Conference on Machine Learning, pp. 961 1624-962.. PMLR, 2020.
Liu, 7.,, Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, 7.,, Lin, S., and Guo, B. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Wang, Y., Zou, D., Yi, J., Bailey, J., Ma. X., and Gu, Q. Im- proving adversarial robustness requires revisiting misclas- sified examples. In International conference on learning representations, 2019.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, 2018
Wei, Z., Wang, Y., Guo, Y., and Wang, Cfa: Class-wiss calibrated fair adversarial training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11133-8201 2023.
J.-.., Wang, X.-D., Zhou, L.-F., and Cheng, K. Adver- sarial examples based on object detection tasks: A survey. Neurocomputing, 519:1 14-111222 126, 2023.
Wu, D., Xia, S.-T., and Wang, Y. Adversarial weight pertur- bation helps robust generalization. Advances in neural information processing systems, $3:2958-2966 2020.
Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., and Anandkumar, A. Diffusion models for adversarial purifi- cation. arXiv preprint arXiv:2205.074600 2022.
Xia, R., Zhou, D., Liu, D., Li, J., Yuan, I.., Wang, N., and
Randen, T. and Husoy, J. H. Filtering for texture classifica- tion: A comparative study. IEEE Transactions on pattern analysis and machine intelligence, 21(4):291-31 1999.
Gao, X. Inspector for face forgery detection: Defending against adversarial attacks from coarse to fine. IEEE Transactions on Image Processing. 2024.
Ren, J., Jiang, X., and Yuan, J. Learning lbp structure by maximizing the conditional mutual information. Pattern Recognition, 48(10):3180-3 10-3319,, 2015.
Xiao, C., Zhu, J., Li, B., He, W., Liu, M., and Song, D. Spatially transformed adversarial examples. In 6th Inter- national Conference on Learning Representations, 2018.
10

---

Phase and Amplitude-wwaee Prompting for Enhancing Adversarial Robustness
Xu, X., Zhang, J., Liu, F., Sugiyama, M., and Kankanhalli, M. S. Enhancing adversarial contrastive learning via adversarial invariant regularization. Advances in Neural Information Processing Systems, 36, 2024.
Yang, X., Gong, Y., Liu, W. Bailey, J., Tao, D., and Liu, W. pemantic-preserving adversarial text attacks. IEEE Transactions on Sustainable Computing, 8(4):583-555, 2023.
Ying, L., Hertzmann, A., Biermann, and Zorin, D. Tex- ture and shape synthesis surfaces. In Rendering Tech- niques 2001: Proceedings of the Eurographics Workshop in London, United Kingdom, June 25-27, 2001 12, pp. 301-312. Springer, 2001.
Zagoruyko, S. Wide residual networks, arXiv preprint arXiv:1605.07146. 2016.
Zhang, G., Zhang, Y., Zhang, Y., Fan, W., Li, Q., Liu, S., and Chang, S. Fairness reprogramming. Advances in neural information processing systems, 35:34347-34362, 2022.
Zhang, H., Yu, Y., Jiao, J., Xing, E., Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Zhang, L., Zhang, L., Mou, X., and Zhang, D. Fsim: A feature similarity index for image quality assessment. IEEE transactions on Image Processing, 20(8):2378- 2386, 2011.
Zhao, K., Kang. Q., Song, Y., She, R., Wang, S., and Tay, W. P. Adversarial robustness in graph neural networks: A hamiltonian approach. Advances Neural Information Processing Systems, 36, 2024.
Zhou, D., Chen, Y., Wang. N., Liu, D., Gao, X., and Liu, T. Eliminating adversarial noise via information discard and robust representation restoration. International Con- ference on Machine Learning, pp. 42517-42530. PMLR, 2023.
Zhou, J., Zhu, J., Zhang, J., Liu, T., Niu, G., Han, B., and Sugiyama, M. Adversarial training with complementary labels: on the benefit of gradually informative attacks. Advances in Neural Information Processing Systems, 35: 23621-23633 2022.
Zhou, Y. Xia, X., Lin, Z. Han, B., and Liu, T. Few-shot adversarial prompt learning on vision-language models. Advances in Neural Information Processing Systems, 37: 3122-3156, 2024.

---

Phase and Amplitude-waare Prompting for Enhancing Adversarial Robustness
A. Preliminary
Notation. We use capital letters like X and Y to represent random variables. Correspondingly, lower-case letters such as and y are presented as the realizations of X and Y. We use ((x, e) denote the neighborhood of z: {e: || ž| <)). where is the perturbation budget. Here, || I| represents the norm, which can be specified as 4 -nom || lloe and L2-norm I I1a. We define f X (1, 2,, ..., C} as a classification function, where the can be parameterized by deep neural network he with the parameter 0.
Problem Setting. In this paper, the task we focus on is the classification under adversarial settings, which means target models may be misled by adversarial noises. We sample natural data {(z,, y:) based on the distribution of (X, Y), where X and Y are the variables of natural instances and their ground-truth labels. Here, (X, E X x (1,,,....c c) and e the number of classes. Given a deep neural network hg and a pair of natural data (x, y), the adversarial example ă is crafted following such a constraint:
ha( + y s.t. | - = S
(1I)
where = 5 and ō represents the adversarial noises. Since our focus is on attacking and defending for images, we utilize the Discrete Fourier Transform (DFT) and its inverse version (IDFT), denoted as F(-) and (-,).,). -). respectively. The phase and amplitude spectra are derived as Φ, = (z) and & (x)). Specifically, we use o.. and &.r to denote the phase and amplitude spectra of a natural image æ, while 62 and &a represent the corresponding spectra of an adversarial example ä. In addition, the process to recover an image from its phase and amplitude spectra is expressed as x = (02. Our goal to design a set of prompts to assist the classification model he in making accurate predictions. These prompts are trained without the need of model retraining. and are further utilized during testing.
B. Prompt Selection Method for Testing from C-AVP
C-AVP (Chen et al., 2023) aims at utilizing pixel domains for prompting. It trains prompt for each class, and designs a prompt selection method that traverses all the prompts from all the classes to get the final predictions for testing on naturally pre-trained models especially for CIFAR-10, which can be formulated as:
p p Pr. i" argmazzechh (Rrees + Ps),
(12)
where p is the selected prompt for the test image Ztest, while p; is the prompt of class i and hig is the output of class i, and denotes the set of classes. Clearly, when the number of classes becomes large, this strategy for testing can easily cause extremely high computational costs. The prompt selection strategy of C-AVP inefficient on numerous classes, and results in Table 4 show baselines with this strategy lose natural accuracy a lot. In comparison, our prompt selection strategy is efficient numerous classes, and our defense with this strategy achieves superior defenses with higher natural accuracy, verifying the superiority of our prompt selection strategy.
C. Experimental Settings
Datasets and Models. In this paper, we consider two popular benchmark datasets CIFAR- 10 (Krizhevsky et al., 2009) and Tiny-ImageNer (Le & Yang, 2015). CIFAR-10 has 10 classes of images with resolution of 32 x 32, which contains 50,000 training images and 10,000 testing images. The larger dataset Tiny- ImageNet has 200 classes with resolution of 64 x 64 and has 100,000 training images, 10,000 validation images and 10,000 testing images. Images in all of these datasets are regarded as natural examples. We normalize all the images into the range of [0, 1]. Data augmentations including random crop and random horizontal flip are performed for all the data in the training stage. For the target model, we use ResNet18 (He et al., 2016) and WideRes.Net8-11 10 (WRN28-10) 10) (Zagoruyko, 2016) for these datasets. We use WRN28- 10, VGG19 (Simonyan & Zisserman, 2014) and a popular vision transformer architecture Swin Transformer (Liu et al., 2021) for evaluating the defense transferability across different models. The Swin Transformer trained following previous studies about evaluating its defense performances (Liu al., 2023).
Attack Settings. We introduce white-box attacks and black-box attacks to evaluate the defense. For white-box attack, we utilize Los -nom AA (Croce& & Hein, 2020), -o-morm C&W (Carlini & Wagner, 2017) and -2-norm DDN (Rony et al., 2019). The iteration number of Ly-norm DDN is set to 20, while that of Lg-norm C&W is 50. The perturbation budget for Loee norm AA is 8/255. For Los norm C&W, the learning rate is 0.01 and the confidence is O. All the attacks mentioned above are set as non-targeted attacks. For black-box attacks, we apply transfer-based attacks under Trge norm AA, L-g-norm
12

---

Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness
DDN using VGG19 the surrogate model and query-based attacks under Square (Andriushchenko al., 2020). The number of queries for Square is set 200.
Defense Settings. We introduce two recently proposed prompt-based defenses C-AVP (Chen et al., 2023) and Freg (Huang et al., 2023) as baselines, which utilize the pixel domain and the frequency domain for prompting respectively. In addition, for the pre-trained models for optimizing and evaluating our prompts, we introduce natural training, AT (Madry et al., 2018), TRADES (Zhang et al., 2019) and MART (Wang et al., 2019). Note that all the pre-trained models are fixed without participating in any prompt training procedure. For the attack during training, we use PGD, where the perturbation budget and perturb step are 8/255 and 10, and the step size is 2/255. We train them using SGD (Andrew & Gao, 2007) for 100 epochs. The initial learning rate is 0.1 with batch size 512 for CIFAR-10, and batch size 256 for Tiny-ImageNet. The initial learning rate is divided by 10 at the 75-th epoch. We set λ 3, X2 400, λ 4 for naturally pre-trained models, and \1 = 1, x = 5000, X3 4 for adversarially pre-trained models.
D. Defending against Black-box Attacks
We perform black-box attacks under transfer-based attacks and query-based attacks. The results are shown in Table 12. is shown that method achieve superior robust performances under black-box settings compared with baselines.
Table Robust accuracy (percentage) of defenses against black-box attacks on CIFAR-10. The target model ResNet18, and the surrogate model is VGG19. we perform AA and DDN the hanse attack strategies.
Defense
None
AA
DDN
Square
NAT
94,83
16.91
53.46
22.10
+Freq
94.50
19.37
53.85
26.43
+C-AVP
92.67
17.27
52.87
22.64
+PAP(Ours)
87.12
51.90
74.79
61.38
E. Defense Transferability to Vision Transformers
We further transfer our prompts trained ResNet18 popular Swin Transformer for evaluating the defense transferability of method. As shown in Table 13, method can be transferred well to vision transformers for improving their robustness, verifying the superior transferability across both convolutional neural networks and vision transformers.
Table 13. Robust accuracy (percentage) of prompts transferred vision transformers, The prompts trained ResNet18, and the vision transformer we introduced is Swin Transformer.
Defense NAT
None
AA
C&W
DDN
88.98
0.00
0.00
0.00
+Freq
82.77
3.47
27.55
15.50
+C-AVP
30.33
9.00
17.28
12.90
+PAP(Ours)
84.82
10.13
77.97
49.15
F. Hyper-parameter Studies
We perform several ablation studies losses with different hyper- syper-parameters as follows. each hyper-parameter, varies within certain range while other hyper- paremeerrss are fixed. It can be seen that the natural and robust accuracies vary under different settings, and the parrmtear set can achieve superior performances in both natural accuracy and robust accuracy.
There exists a trade-off problem in our method. As shown in Figure 5, for naturally pre-trained models, the natural accuracy increases while the robust accuracy drops as X or 22 increases. As shown in Figure for adversarially pre-trained models, when A2 varies from 0 to 5000, the trade-off problem exists explicitly. Overall, the hyper--aramnn parameters we set achieve superior performances in both natural and robust accuracies.
13

---

Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness
50
100
50
100
50
100
45
90
45
90
45
90
40
40
80
30
60
30
60
30
60
25
50
25
50
25
50
15
30
10
20
10
20
10
20
10
>
10
200
2/2
400
600
accurncy
Natural accuracy
accuracy
Natural uccuracy
Robust accuracy
accuracy
Figure 5. The impact of losses with different varies within certain range while other
naturally pre-trained ResNet18 in CIFAR-10 10. For cach hyper-pann parameter, are fixed. We show the natural accuracy and robust accuracy against AA.
75
90
75
90
75
90
72
88
72
88
72
88
66
84
84
66
84
63
82
63
82
63
82
60
80
60
80
60
80
57
78
54
76
54
76
54
76
51
74
51
74
51
74
48
72
72
48
72
20
5000
2-
10000
15000
12
Robust accuracy
Natural accuracy
accurncy
accuracy
Robust accuracy
Natural accuracy
Figure 6. The impact of losses with different parametas adversarially pre-trained ResNet18 in CIFAR-10. For each hyper- parameter, varies within a certain range while other hyper-parameters are fixed. We show natural accuracy and robust accuracy against AA.
G. Visualizations of Prompted Images
We present additional visualized results of prompted images using our prompts, which are presented as follows. Here, following previous works (Elsayed et al., 2018; Tsai et al., 2020; Zhang et al., 2022), C-AVP performs prompting in the pixel space by adding random noises the surrounding area inside the image, only keeping the square area in the center unchanged. Therefore, C-AVP is only frame. It can be that our method retains complete and natural semantic patterns after prompting.
C-AVP performs prompting by adding noises around the image in the pixel domain, while Freq performs prompting on the high-frequency domain. They both train their prompts without considering their disruptions on the natural semantic patterns. In comparison, our method construct prompts on more specific semantic patterns, training them to enforce the prompted images to as similar as possible to corresponding natural images. This can preserve more natural semantic patterns as shown in Figure 4, 7 and
H. Stability in Natural Accuracy
As whole, our method performs more stably in natural accuracy. As shown in Section 4, basclines lose more natural accuracy under many cases, such as the worse transferability and performances under adaptive attacks of C-AVP and the natural accuray drop of Freq shown in Table under adversarially pre- praitrained models. In comparison, our defense remains high natural accuracy in all of these cases, verifying the stability our defense.
14

---

Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness
I. Effectiveness on C&W
C&W method generates adversarial perturbations by performing optimizations in the pixel domain. Differently, our approach additionally considers the frequency domain. disentangles the frequency domain information and leverages the amplitude and phase spectra as way focus more finely on important structural semantics and textures, which are not covered the compared baselines. Therefore, our method provide effective defense against perturbations generated by C&W. W.
Input examples
Freq
C-AVP
PAP(Ours)
Figure 7. Visualizations of prompted images for input examples CIFAR-10. The target model is naturally pre-trained ResNet18. For each pair images, the left part denotes the prompted image, while the right part denotes the difference heatmap compared to the original input (i.e., adversarial) example.
Input examples
jkke
Freq
C-AVP
PAP(Ours)
oke
Figure 8. Visualizations of prompted images for input examples on Tiny- ImageNet.. The target model is naturally pre-trained WRN28- 10. each pair of images, left part denotes the prompted image, while right part denotes the difference heatmap compared to the original input (ie., adversarial) example.
15


Reviews:
[{"review_id": "XSmSTlzSlC", "reviewer": "Reviewer_pd1Q", "summary": "This paper proposes a defense strategy based on prompting on structures and textures, with appropriate weights adjusted by their influences on robustness for incorporating their benefit for defenses. It achieves superior defense performances on general and adaptive attacks and defense transferability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and constructive suggestions. The responses to your concerns are as follows:\n\n**Discussions on the trade-off problem.** Our method has a trade-off problem under different hyper-parameters. On naturally pre-trained models, when λ1 increases, the natural accuracy increases while the robust accuracy drops explicitly. As λ2 increases, the robust accuracy drops a lot while the natural accuracy does not change too much for the naturally pre-trained models. On adversarially pre-trained models, the trade-off problem only exists when λ2 varies. Overall, the selected hyper-parameters achieve superior performances in both natural and robust accuracies. We would state it in the updated version:\n\n*\"There exists a trade-off problem in our method. As shown in Figure 5, for naturally pre-trained models, the natural accuracy increases while the robust accuracy drops as λ1 or λ2 increases. As shown in Figure 6, for adversarially pre-trained models, when λ2 varies from 0 to 5000, the trade-off problem exists explicitly. Overall, the hyper-parameters we set achieve superior performances in both natural and robust accuracies.\"*\n\n**Analyses of visualizations of prompted images.** C-AVP performs prompting only by adding noises around the image in the pixel space to mitigate the negative effects of adversarial perturbations. Frequency Prompting method directly adds frequency prompts on the high-frequency domain. They both construct and train their prompts without considering their disruptions on the natural semantic patterns in the pixel space. In comparison, our method constructs prompts on the more specific semantic patterns (i.e., textures from amplitude spectra and structures from phase spectra), and trains them using a loss which enforces the prompted images to be as similar as possible to the corresponding natural images in the pixel space. To this end, our method preserves more natural semantic patterns after prompting compared with baselines. We would state it in the updated version:\n\n*\"C-AVP performs prompting by adding noises around the image in the pixel domain, while Freq performs prompting on the high-frequency domain. They both train their prompts without considering their disruptions on the natural semantic patterns. In comparison, our method construct prompts on more specific semantic patterns, training them to enforce the prompted images to be as similar as possible to corresponding natural images. This can preserve more natural semantic patterns as shown in Figure 4, 7 and 8.\"*\n\n**Fairness of the comparison with universal prompts.** The data-prompt mismatching loss indeed cannot be performed on universal prompts. For fairness, we remove the data-prompt mismatching loss on universal prompts, and other training settings for universal prompts are the same as those from our method. We would state it in the updated version:\n\n*\"For fairness, the data-prompt mismatching loss is removed on universal prompts, and other settings for universal prompts are the same as those from our method.\"*\n\n**Typos.** We are sorry for this typo. We would correct the typo: \n\n*“Clearly, when the number of classes becomes large, this strategy for testing can easily cause extremely high computational costs.”*\n\n**Settings of adaptive attacks.** For adaptive attacks on training and testing, the perturbation budget is 8/255 and the step size is 2/255. The iteration number is 10 for training, and 20 and 40 for testing.\n\n**The integrity of spectra after prompting.** We conduct a reconstruction loss to train the prompts, enforcing the prompted images to be as similar as possible to the corresponding natural images in the pixel space. Therefore, our method does not disrupt the integrity of these spectra, which can be shown in the Figure 4, 7 and 8.\n\nWe sincerely hope our answers can solve your concerns and obtain your increase in the score.", "claims_and_evidence": "Yes, the claims made are all supported by clear evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method with the criteria makes sense for the problem and application.", "theoretical_claims": "Yes, I checked the correctness and found that they are all correct.", "experimental_designs_or_analyses": "Yes, I checked the validity of these designs and consider they are reasonable and sound.", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to previous analyses about the semantic patterns of phase and amplitude spectra. They are utilized in the submission for mitigating the negative effects on specific semantic patterns to enhance the robustness of prompt-based defenses.", "essential_references_not_discussed": "No, there are not related works that are essential but are not discussed", "other_strengths_and_weaknesses": "Strengths\n+ The framework and its corresponding motivations are clearly presented, with precise expressions of formulas and figures.\n\n+ Given the inefficiency of traditional defenses, the area the authors focus on is novel, where the use of prompting on specific semantics with appropriate weights offers promising directions for exploration.\n\n+ The experiments presented in the manuscript comprehensively validate the effectiveness of the method from multiple perspectives.\n\nWeaknesses\n- Authors should discuss the potential problem of trade-off between natural accuracy and robustness under the proposed method. The proposed defense improves robustness by a large margin according to the presented results, while the trade-off exists according to hyper-parameter studies, which deserves to be discussed briefly for the integrity of logic.\n\n- The analyses of visualizations of prompted images may need to be cleared further. The baselines perform prompting in a way that is different from the proposed defense, and analyzing their limitations in preserving natural semantic patterns is somewhat necessary.\n\n- The fairness of the ablation study comparing with universal prompts needs to be addressed. When training universal prompts, the data-prompt mismatching loss cannot be applied, and this may need to be removed for a fair comparison. Please clarify this point.", "other_comments_or_suggestions": "A minor grammar mistake exists in Appendix B, where the phrase “the number of class” may need to be modified as “the number of classes”.", "questions_for_authors": "- In the experiments, I am concerned about the attack settings of adaptive attacks, considering the fairness of evaluations.\n- Directly applying prompts to the amplitude and phase spectra is a novel approach. However, could this method potentially disrupt the integrity of these spectra?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ulqlQw1BO0", "reviewer": "Reviewer_fSJb", "summary": "This work proposes a prompting method for defense, through training prompts for each class using specific semantic patterns including structures and textures based on the Fourier Transform, which successfully defends against various general and adaptive attacks.\n\n## Update After Rebuttal\n\nThe authors have addressed all my concerns comprehensively in their rebuttal. In particular, the authors clarify the stability of natural accuracy under different scenarios (e.g., transferability, adaptive attacks) with additional comparisons to baselines and reorganized hyperparameter settings for better readability. After reviewing the rebuttal, I maintain this score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive suggestions. Your comments are important to our work, and we address them as follows:\n\n**Stability in natural accuracy.** Our method lose a few natural accuracy when performing defenses. However, as shown in the Section 4, baselines lose more natural accuracy under various scenarios, such as the performances of C-AVP under transferability evaluations and those of Freq in Table 3. In comparison, our method remains high natural accuracy in these scenarios, achieving more stable performances in natural accuracy. We would state it in the updated version:\n\n*\"As a whole, our method performs more stably in natural accuracy. As shown in Section 4, baselines lose more natural accuracy under many cases, such as the worse transferability and performances under adaptive attacks of C-AVP and the natural accuray drop of Freq shown in Table 3 under adversarially pre-trained models. In comparison, our defense remains high natural accuracy in all of these cases, verifying the stability of our defense.\"*\n\n**The introduction of experimental settings.** We are sorry for such a presentation that is not easy to read. We would introduce all of them in the section of experimental settings clearly in the updated version:\n\n*\"We set λ1=3, λ2=400, λ3=4 for naturally pre-trained models, and λ1=1, λ2=5000, λ3=4 for adversarially pre-trained models. The threshold τ is set as 0.1, and we adjust the weights of amplitude-level prompts every 5 epochs.\"*\n\n**The consistency of the phrase.** We are sorry for the inconsistency. We would correct the phrase:\n\n*“To this end, we apply Gaussian Blur on the test image for evaluations.”*\n\n**Settings of adaptive attacks.** During training and testing, for other settings of adaptive attacks, the perturbation budget is 8/255, and the step size is 2/255.\n\n**Influences of the data-prompt mismatching loss.** The data-prompt mismatching loss does not contradict the classification loss. As shown in Figure 5, when the λ3 varies from 0 to 4, both the natural and robust accuracies increase by a large margin. Therefore, training prompts using these losses simultaneously can achieve superior performances in both natural and robust accuracies.\n\n**The settings of Gaussian Blur in the ablation study.** For the Gaussian Blur performed in the ablation study, the kernel size is set as 3, and the standard deviation is sampled randomly from 0.1 to 2.0. The Gaussian Blur is conducted on the test image for evaluations.\n\nWe sincerely hope our answers can solve your concerns and obtain your increase in the score.", "claims_and_evidence": "Yes, these claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method and criteria make sense for the problem or application.", "theoretical_claims": "Yes, I checked the correctness of them and found that there are no issues among them.", "experimental_designs_or_analyses": "Yes, I checked the soundness of all the experimental designs or analyses and concluded that they are sound and valid.", "relation_to_broader_scientific_literature": "The key contributions of prompting on phase and amplitude spectra are related to the previous findings that analyzed the phase and amplitude spectra of images hold structures and textures respectively.", "essential_references_not_discussed": "No. There aren’t related works that are essential but are not discussed in the paper.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The proposed defense strategy firstly exploits visual prompts from specific phase and amplitude spectra, which explores a promising direction for efficient defenses utilizing specific semantic patterns. \n2. Experiments from various viewpoints including adaptive attacks and transferability evaluation verify the effectiveness of the proposed method in defenses. \n3. The training and testing procedures are introduced with a reasonable and sound logic, where they are analyzed and proven to be effective by sufficient defense evaluations and ablation studies.\n\n**Weaknesses:**\n\n1. The stability of the proposed method in natural accuracy under various scenarios needs to be cleared. It is clear that the method provides great robust gains while reducing a few natural performances which is mentioned as limitations. In fact, baselines lose more natural accuracy when evaluating on transferability and adaptive attacks, which needs to be discussed. \n2. The threshold, hyper-parameters and the frequency of adjusting weights are presented in a somewhat dispersed manner. Maybe they can be centered in experimental settings so that readers can notice these settings efficiently.", "other_comments_or_suggestions": "A few typos about the consistency of phrases need to be considered, i.e., “tested image” in the ablation study which is “test image” in the context.", "questions_for_authors": "1. What are the experimental settings for adaptive attacks when training and testing, except for the number of iterations? \n2. I wonder if the designed data-prompt mismatching loss contradicts the classification loss considering the different assignments of prompting of different losses for training. \n3. What are the settings of gaussian blur performed in the ablation studies?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SN39oMyZGt", "reviewer": "Reviewer_9KjN", "summary": "This work exploits a prompt-based defense using specific texture and structure patterns, and proposes to incorporate these prompts with appropriate prompting weights according to their effects on robustness, which enhances the robustness in various scenarios with superior transferability across various networks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. The responses to your concerns are as follows:\n\n**The superiority of our method.** Our method indeed does not outperform “Freq” by a large margin in a few scenarios. However, as shown in the Table 3, 4, 6, 7, 12 and 13, Freq sacrifices natural accuracy by a large margin in many cases. In comparison, the natural accuracy drop of our method is fewer, and our method achieves superior defenses in almost all of the defense cases.\n\n**The superiority of the prompt selection strategy.** The prompt selection method of C-AVP will become extremely inefficient on numerous classes. Meanwhile, as shown in Table 4, baselines using this strategy sacrifice natural accuracy by a large margin, further verifying its limitations on the model’s performances. Instead, our prompt selection strategy is efficient on numerous classes, and our defense using this strategy achieves superior defense without losing natural accuracy too much. We would state it in the updated version:\n\n*\"The prompt selection strategy of C-AVP is inefficient on numerous classes, and results in Table 4 show baselines with this strategy lose natural accuracy a lot. In comparison, our prompt selection strategy is efficient on numerous classes, and our defense with this strategy achieves superior defenses with higher natural accuracy, verifying the superiority of our prompt selection strategy.\"*\n\n**Minor mistakes.** We are sorry for these mistakes. We would correct them:\n1) *“Nat. Pha./Amp. indicates we replace phase/amplitude spectra with corresponding natural spectra.”*\n2) *“Motivated by the above studies, we propose a Phase and Amplitude-aware Prompting (PAP) defense mechanism, which constructs phase and amplitude-level prompts to stabilize the model’s predictions during testing.”*\n\n**Training strategy of baselines.** For baselines Freq and C-AVP, we train them following the settings from their original papers without any modification for a fair comparison.\n\n**The superiority compared with universal prompts.** The defense superiority of our method still exists against attacks which are different from AA. As shown in the table below, our method achieves superior defenses against various attacks compared with universal prompts.\n|  | None | L∞-norm PGD | L2-norm PGD |\n|---|:---:|:---:|:---:|\n| NAT | **94.83** | 0 | 0.37 |\n|   +Universal | 87.54 | 30.23 | 57.05 |\n|   +PAP(Ours) | 87.12 | **35.45** | **58.94** |\n\nWe sincerely hope our answers can solve your concerns and obtain your increase in the score.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method and evaluation criteria all make sense for problems and applications.", "theoretical_claims": "Yes, I checked the correctness of them, and they are all reasonable and correct without any problem.", "experimental_designs_or_analyses": "Yes, I checked the validity of these designs and analyses, and consider that they are all sound.", "relation_to_broader_scientific_literature": "The key contributions of this paper are related to the findings of the semantic meanings about phase and amplitude spectra, and they are utilized as prompting, which is more innovative and efficient compared with previous adversarial training and denoising methods.", "essential_references_not_discussed": "No, there are not related works that are essential but are not currently cited or discussed.", "other_strengths_and_weaknesses": "Strengths\n+ Novelty: Introducing the popular field of prompts provides a novel and feasible idea for defense efficiently compared with adversarial training and denoising methods.\n+ Incorporations of different semantic patterns: Incorporating prompts from different semantic patterns based on their influences on robustness exploits the benefits of these patterns innovatively.\n+ Transferability: The proposed defense achieves superior transferability across convolutional neural networks and vision transformers compared with previous prompt-based defenses, verifying its practicality.\n\nWeaknesses\n- Few marginal results: In the experiments, while the proposed defense improves the robustness against various attacks a lot, it seems that the proposed method does not explicitly outperform the baseline named “Freq” in a few scenarios.\n- Explanations about the superiority of the prompt selection strategy: The drawback of the prompt selection strategy of C-AVP for testing needs to be clarified comprehensively. It is introduced that this strategy is inefficient on numerous classes, and baselines using it sacrifice natural accuracy by a large margin. Bringing more clarity here can help further verify the superiority of the proposed method.", "other_comments_or_suggestions": "(1) In the caption of Table 1, the word “indicate” after “Nat. Pha./Amp.” should be “indicates”.\n(2) In the 4-th paragraph of Introduction, it mentions “phase and amplitude prompts”, which is mentioned as “phase and amplitude-level prompts” in the context.", "questions_for_authors": "(1) How does the Freq and C-AVP train for comparing with our method?\n(2) It has been shown that the proposed method for learning prompts for each class achieve better results against AA compared with universal prompts. Does this superiority still exist against other attacks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UzStVQDTrd", "reviewer": "Reviewer_h61T", "summary": "This paper shows another defense based on visual prompting. They extended previous defenses applying the Fourier transform and integrating this in their defense design. \nThis method can outperform previous methods and showed its effectiveness against adaptive attacks.\nThe limitations could have been better discussed. \n\nNevertheless, in overall, I can recommend this paper.\n\n\n## update after rebuttal\nNone", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your great efforts spending on reviewing our paper. Your comments are important to the improvement of our work, and we address them as follows:\n\n**The clarity on limitations.** We are sorry for the lack of clarity on limitations. The main limitation of our method is the sacrifice of natural accuracy. We would provide a clear introduction to this issue in a separate section. In addition, due to the limitation of computational resources, we do not perform evaluations on ImageNet and feel sorry for that. However, we conduct experiments on Tiny-ImageNet which has been widely used. Tiny-ImageNet-200 is larger and has a larger resolution than CIFAR-10, with more numbers of classes than those of ImageNet-100. Results show that our method achieves superior performances, leading us to believe that our method can also work well on ImageNet. We would state it in the updated version.\n\n**The selection of attacks.** Table 2 aims at verifying the different impacts of amplitude-level and phase-level prompts on robustness. In Table 2, we use PGD which is a popular method for evaluating the defense, to efficiently reflect the impacts of them on robustness. Table 3 and other evaluations in Section 4 aim at comprehensively evaluating the defense effectiveness of the proposed method. Therefore, we use the stronger AutoAttack which contains PGD to evaluate the proposed method comprehensively.\n\n**Typos.** We are sorry for these typos. We would correct them as follows:\n1) Ln 12: *“Deep neural networks are found to be vulnerable to adversarial perturbations.”*\n\n2) Ln 71: *“White-box attacks like Projected Gradient Descent (PGD) attack (Madry et al., 2018), AutoAttack (AA) (Croce & Hein, 2020), Carlini&Wagner (C&W) (Carlini & Wagner, 2017) and Decoupling Direction and Norm (DDN) (Rony et al., 2019) craft noises through accessing and utilizing models’ intrinsic information like structures and parameters.”*\n\n3) Ln158: *“However, existing prompt-based defenses focus on mixed patterns like pixel or frequency information, which cannot capture specific patterns like structures and textures (Ying et al., 2001; Ren et al., 2015).”*\n\n4) Equation 2 and 3: We are sorry for the waste of space here. We would move the \"Classification Loss\" section and these equations into the Appendix in the updated version.\n\n5) Ln325: Yes, the default perturbation budget for L-2 norm is 0.5. We would supplement it in the updated version:\n\n*\"The perturbation budget for L∞-norm AA is 8/255. And the perturbation budget for L-2 norm attacks is 0.5.\"*\n\n**Explanations of C-AVP.** Following previous works [1-4], C-AVP performs prompting in the pixel space by adding random noises to the surrounding area inside the image, only keeping the square area in the center unchanged. Therefore, C-AVP is only a frame. We would state it in the updated version.\n\n[1] Visual prompting: modifying pixel space to adapt pre-trained models. arXiv:2203.17274, 2022.\n\n[2] Adversarial reprogramming of neural networks. arXiv:1806.11146, 2018.\n\n[3] Transfer learning without knowing: reprogramming black-box machine learning models with scarce data and limited resources. ICML, 2020.\n\n[4] Fairness reprogramming. NeurIPS, 2022.\n\n**Rationale behind suggesting contrastive learning.** In the defense area, Contrastive Learning is a useful method for mitigating the trade-off problem between natural and robust accuracies, by learning the invariant natural semantic information between natural and adversarial examples through contrastive losses [5-7]. Therefore, we suggest Contrastive Learning and hope it can help break the limitation of our method about sacrificing natural accuracy when performing defenses. We would state it in the updated version.\n\n[5] Adversarial self-supervised contrastive learning. NeurIPS, 2020.\n\n[6] Robust pre-training by adversarial contrastive learning. NeurIPS, 2020.\n\n[7] Enhancing adversarial contrastive learning via adversarial invariant regularization. NeurIPS, 2024.\n\n**Effectiveness on C\\&W.** C\\&W method generates adversarial perturbations by performing optimizations in the pixel domain. \n Differently, our approach additionally considers the frequency domain.  It disentangles the frequency domain information and leverages the amplitude and phase spectra as a way to focus more finely on important structural semantics and textures, which are not covered in the compared baselines. Therefore, our method can provide a more effective defense against perturbations generated by C\\&W. We would supplement this statement in the updated version.\n\nWe sincerely hope our answers can solve your concerns and obtain your increase in the score.", "claims_and_evidence": "They extended a prompt defense with the Fourier transform and designed their own optimization algorithm. \nIn table 8, 9, 10, and 11, they showed empirically the effectiveness.", "methods_and_evaluation_criteria": "The proposed method has been adequately benchmarked. The scarification of the standard accuracy could be demonstrated more clearly.", "theoretical_claims": "This paper is more closely aligned with the field of applied science, wherein an algorithm is identified and a loss function is formulated. Additionally, the Fourier transform and its inverse are applied to the entire image.\nThe request for additional theoretical proofs is not within the scope of the author's expertise.", "experimental_designs_or_analyses": "The classification models and datasets are sufficient, and the experimental design makes sense and also covering adaptive adversarial examples.\n\nI do not understand, why C&W becomes so good in Table 6. When I look at Figure 2 in this paper: https://arxiv.org/pdf/2112.01601 then the FFT analysis does not seem so strong.  Any explanation?", "relation_to_broader_scientific_literature": "Prompting is a common technique for fine-tuning, particularly within the domain of NLP.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strength:\n - Effective method\n - Many experiments\n\nWeaknesses:\n - The limitations could have been presented with greater transparency. In order to locate them, it was necessary to conduct a search, particularly with regard to the scarification of the standard accuracy. Additionally, the potential impact on larger datasets, such as ImageNet, could have been examined.\n -  The rationale behind the selection of adversarial attacks from a multitude of options remains opaque. The evaluation of PGD, while not an invariable practice, does occur on occasion. It is postulated that this practice will not engender a substantial disparity in the context of AA.", "other_comments_or_suggestions": "ln12 noises -> Better perturbation\nln71 PGD -> better write PGD attack. PGD is just the optimization algorithm. And over time in related work - people use it wrongly.\nLn158 -> cite \n- https://cse.buffalo.edu/~jsyuan/papers/2015/Learning%20LBP%20Structure%20by%20Maximizing%20the%20Conditional%20Mutual%20Information.pdf\n- https://cims.nyu.edu/gcl/papers/ying2001tss.pdf\n\nIs it necessary to give equation 2 and 3 so much space?\nLn325: L2 norm is not described the per budget?  0.5?", "questions_for_authors": "1. Could you kindly explain your rationale for choosing PGD in table 2 over table 3? \n2. Could you please explain why C-AVP is only a frame? It seems that on the CIFAR-10 dataset, the perturbation is in the middle of the image, and your FFT is applied globally. \n3. Regarding limitations, it seems standard accuracy is being compromised. Could you please elaborate on the rationale behind suggesting contrastive learning in this context? \n4. Could you please elaborate on the rationale behind the defense on C&W being so effective?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes PAP (Phase and Amplitude-aware Prompting), a novel prompting-based defense that enhances adversarial robustness by leveraging class-specific prompts constructed from phase (structure) and amplitude (texture) spectra. Prompt weights are adaptively optimized during training based on robustness, and selected at inference using predicted labels. The method achieves strong performance across various attacks and datasets, demonstrating improved robustness and transferability with greater efficiency and interpretability.

The reviewers unanimously recommended acceptance and praised the paper's originality, strong empirical results, and well-structured methodology. Key strengths include the effective use of semantically meaningful prompts, solid evaluations against general and adaptive attacks, and fair comparisons to baselines. Minor concerns were raised about clarity regarding natural accuracy trade-offs, adaptive attack configurations, and the organization of hyperparameter details. These were adequately addressed in the rebuttal.
Decision: Accept (poster)