On Trace of PGD-Like Adversarial Attacks
Authors: Mo Zhou
Vishal Patel
Venue: NeurIPS 2022 Submitted (2022)
Keywords: adversarial attack
adversarial attack detection
adversarial defense

Abstract: Adversarial attacks pose safety and security concerns for deep learning applications.  Yet largely imperceptible, a strong PGD-like attack may leave strong trace in the adversarial example.  Since attack triggers the local linearity of a network, we speculate network behaves in different extents of linearity for benign examples and adversarial examples.  Thus, we construct Adversarial Response Characteristics (ARC) features to reflect the model's gradient consistency around the input to indicate the extent of linearity.  Under certain conditions, it shows a gradually varying pattern from benign example to adversarial example, as the later leads to Sequel Attack Effect (SAE).  ARC feature can be used for informed attack detection (perturbation magnitude is known) with binary classifier, or uninformed attack detection (perturbation magnitude is unknown) with ordinal regression.  Due to the uniqueness of SAE to PGD-like attacks, ARC is also capable of inferring other attack details such as loss function, or the ground-truth label as a post-processing defense.  Qualitative and quantitative evaluations manifest the effectiveness of ARC feature on CIFAR-10 w/ ResNet-18 and ImageNet w/ ResNet-152 and SwinT-B-IN1K with considerable generalization among PGD-like attacks despite domain shift.  Our method is intuitive, light-weighted, non-intrusive, and data-undemanding.

On Trace of PGD-Like Adversarial A Attacks
Anonymous Author(s)
Affiliation
Address
email
Abstract
Adversarial attacks pose safety and security concerns for deep learning applications. Yet largely impenceptible, strong PGD-like attack may leave strong trace in the adversarial example. Since attack triggers the local linearity of a network, we speculate network behaves in different extents of linearity for benign examples and adversarial examples. Thus, we construct Adversarial Response Characteristics (ARC) features reflect the model's gradient consistency around the input indi- cate the extent of linearity. Under certain conditions, it shows gradually varying pattern from benign example to adversarial example, as the later leads to Sequel Attack Effect (SAE). ARC feature can be used for informed attack detection (pertur- bation magnitude is known) with binary classifier, (perturbation magnitude is unknown) with ordinal regression. Due to the unique- ness of SAE PGDD-like attacks, ARC is also capable of inferring other attack details such as loss function, or the ground-truth label as post-processing defense. Qualitative and quantitative evaluations manifest the effectiveness of ARC feature CIFAR-10 w/ ResNet-18 and ImageNet w/ ResNet- 152 and SwinT-B-INIK with considerable generalization among PGD-like attacks despite domain shift. Our method is intuitive, light-weighted, non-intrusive, and data-undemanding.
3
10
uninformed attack detection
11
12
18
14
16
Introduction
18
Recent studies have revealed the vulnerabilities of deep neural networks by adversarial attacks 1 2 where undesired output (e.g. misclassification) could be incurred by an imperceptible perturbation added network input, posing safety and security concerns for respective applications. In the literature, PGD-like attacks, including BIM (. PGD [2], MIM [3 and APGD [4] are strong and widely used. Yet, such strong attack may also leave strong trace in its result, as does in the feature maps
20
21
22
23
24
[5]. Consider an extremely limited setting given an already trained deep neural network and merely a tiny set (e.g., 50) of training data, without any change in architecture or weights, nor any auxiliary deep networks, can we still identify any trace of adversarial attack?
25
28
Recall that FGSM 66, the foundation of PGD-like attacks, attributes network vulnerability "local 28 linearity" being easily triggered by adversarial perturbations. Thus, we conjecture that a network
29 behaves higher extent of linearity adversarial examples than benign (i.e., unperturbed) ones.
30 With the first-order Taylor expansion of a network, "local linearity" implies high gradient proximity
31 in the respective local area. Thus, can select series of data points with stable pattern near the
32 input as exploitation vectors using BIM [1] attack, and then compute the model's 's Jacobian matrices
33 with respect to them. Next, the Adversarial Response Characteristics (ARC) matrix constructed
34 from these Jacobian matrices reflecting the gradient direction consistency across all exploitation
35 vectors. Different from benign examples, PGD-like attacks will trigger Sequel Attack Effect (SAE), leaving higher values in the ARC matrix and hence reflecting higher gradient consistency among exploitation vectors around the input. Visualization results suggest SAE gradually varying pattern with perturbation magnitude increasing, indicating feasibility of attack detection.
37
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Input Image cotteer beiigaidhennssl
Repert for and golla all coenc simbbrtt murrans
ARC Matrix
Fazmpple Seatier Por
Take Jacobians Trooitape (8,00
ARC Vector
Palastaa (A,
TRciEa nous vall
Aeversanial
187-177-11
Examples (T (8)
Exploitation Vectors (pentaled sanples cocated BDM Extepe)
Jacoktan and stack losgathe:
CIFAR- 1-1 RocNeer
Res 182
ImugerNes
So
Figure 1: Diagram for computing the ARC matrix and the ARC vector. They reflect the model's gradient consistency within local linear area around the input indicate the extent of linearity. Shallow network like ResNet- 18 shows higher linearity to benign examples, while deeper networks like ResNet- and SwinT-B-INLL show lower linearity.
39 The ARC matrix can be simplified into the 2-D ARC vector by fitting a Laplacian function due to their resemblance, in order to make subsequent procedure simple to interpret. The ARC vector can be used for informed attack detection (the perturbation magnitude & is known) with an SVM-based binary classifier; for uninformed attack detection (the perturbation magnitude is unknown) with an SVM-based ordinal regression model. The SAE is the unique trace of PGD-like attacks. Due the uniqueness SAE to PGD-like attacks, once the attack is detected, we can also infer some attack details including the attack loss function, or the ground-truth label used during the attack as a post-procsssing defense method.
41
43
44
45
46
47
We evaluate our method on CIFAR- 10 7 with ResNet- 18 [8) and ImageNet [ with ResNet- 152 [ and SwinT-B-Ii K-IK [10). Qualitative and quantitative experimental results manifest the effectiveness of our method in identifying SAE, the unique trace of PGD-like attacks for attack detection, which also possess considerable generalization capability (despite domain shift among PGD-like attacks) even if training data only involves few benign and adversarial examples from BIM attack.
49
51
52
Contributions. We present the ARC features to identify the unique trace, i.e., SAE of PGD-like attacks from adversarially perturbed inputs. It can used for informed/uniifformedd attack detection and inferring attack details (including correcting prediction). Through the lens of ARC feature (reflecting network's s gradient behavior), we also obtain insights on why networks are vulnerable, well why adversarial training works well defense. Although our method is only sensitive to PGD-like attacks, it is (1) light-weighted (requires no auxiliary deep model); (2) non- con-intrusive (requires change to the network architecture or weights); (3) data-undemanding (can generalize with very few samples). Such a problem setting is extremely limited, requiring strong cues to solve.
53
54
55
57
59
Adversarial Response Characteristics & Sequel Attack Effect
60
A neural network f(-) maps the input R.M into pre-sofimax output E R., where the maximum element after softmax corresponds to the class prediction Ö(x). which is expected to match with the ground truth e(x). Then, typical adversarial attack [ [ aims to find an imperceptbble adversarial perturbation r E TR. M that induces misclassification, i.e., arg max,, fn (xr + r) * c(x)
82
64
where |||p c, x +r E MM and f.(() is the n-th element of vector function f(-).
66 According to FGSM (6, the neural network is vulnerable because the "locally linear" property being triggered by the attack. Thus, we assume that the neural network f(.) behaves relatively non-linear against benign examples, while relatively linear against adversarial examples. Then, f()) can be approximated by the first-order Taylor expansion around either benign or adversarial sample ă:
68
tr. f fn) + 5t f Vn E (1,,,...,N).
(1)
70 where is small vector exploiting the local area around the point ã, and the gradient vector
71 f-)(-) is the n-th row of the Jacobian f.) of size NxM. We name the twice- wiice-perturbed
72 as "exploitation vector". This equation means in order to reflect linear behaviour, the first-order
73 gradient V (-(()) is expected to remain in high consistency (or similarity) the local area regardless
of 5. In contrast, when the input ē is not adversarial (r 0), neither Taylor approximation nor the
75 gradient consistency is expected hold. Next, the gradient consistency will be quantized to verify conjecture, and reveal difference between benign and adversarial inputs.

---

1,00
8.90
A6Om 2/235
10 ARCV 2/25
ARCm 4255
ABCV
10
20
ARCm 8255
18 20
20
ARCm c-16/25
ARCV 19/255
20
10
0.5
2.
3.5-
9.-
AROIS <- 20255
ARCV 2/255
ARCin 4255
ARCx 4/255
ARCm 8255
ARCV <- $255
ARCm 16/25
ABCV 16/255
1.00
1.3
2.0
10"
0.22
2.--
646-
Figure 2: The ARC features (i.e. ARC matrix/vector) of adversarial examples created by the BIM attack. 1st row: R8 on CIFAR-10 10; 2nd row: ResNet-152 on ImageNet; 3rd row: SwinT-B-INN on ImageNet. Blue and red dots in the scatter plots correspond to the benign and adversarial examples, respectively. The cluster centers of the ARC vector correlates with the perturbation magnitude
77 Adversarial Response Characteristies (ARC). Using random noise ò does not lead stable
78 pattern of change in a series of exploitation vectors (á + 6, !.u.1..... Instead, we use Basic Iterative
79 Method (BIM) [ to make f-)) more linear starting from æ, which means to "continue" the attack
80 I is already adversarial, or "restart" otherwise. However, the ground-truth label for an arbitrary ø is
81 unknown. Since PGD-like attacks tend to make the ground-truth least-likely based on our observation. we treat the least-likely prediction ((x) the label. Then, the BIM iteratively maximizes the cross
83 entropy loss Len(E + 6, c(x)) via projected gradient ascent as
8,+1 + Clip, 6, + sign V LCE(E + 8,, (x))1). t= 1, 1,......,
(2)
where (- clips the perturbation to the L, bound centered at Ø, and 5 0. If the input é is benign, then the network behaviour is expected to changed from "very non-linear" to "somewhat- 86 linear" during the process; if the input z is already adversarially perturbed, then the process will "continue" the attack, making the model even more "linear" we call this Sequel Attack Effect (SAE).
85
87
88 To quantize the extent of "linearity", we measure the model's gradient consistency across exploitation vectors with cosine similarity. For each (-). we construct a matrix Sn of shape (T+1,T+I):
89
(i,) cos fn 5,), fa 5,)]. Vi,j 0,
As the model f(-) becoming more "linear" to the input (higher gradient consistency), ) the off-diagonal values in S, is expected gradually increase from the top-left to the bottom-right corner. Note that 92 the attack may not necessarily make all (.) behave linear, so we select the most representative cosine
90
91
matrix with the highest mean as the ARC matrix: S. A S..,, where n² arg max,, Ei, (i.i)
88
94
Due to the resemblance of the ARC matrix to the Laplacian function with matrix diagonal being 95 the center, we simplify it into a two-dimensional ARC vector (A, by fitting L(i, j; A,
96 A exp(- /l//)) with
Beennberg-Maqquarre algorithm [II, where i.j are matrix row and column indexes, while A and a are function parameters. For brevity, we abbreviate the ARC matrix as "ARCm", and the ARC vector as "ARCv". The process for computing them is summarized in Fig.
97
98
Visualizing Sequel Attack Effect (SAE). We compute ARCm based on some benign examples using T-48, as shown in Fig. [ The trend of being gradually "linear" (higher cosine similarity) along the diagonal is found across architectures. Thus, SAE is similar to "continue" attack from halfway on the diagonal in such large ARCm. As illustrated in Fig. 2 already adversarially perturbed input (using BIM) leads to larger cosine similarity the very first exploitation vectors as perturbation magnitude E increases from 0 to 16 16/255. Meanwhile, the cluster separation for ARCv is more and more clear. Thus, a clear and gradually changing pattern can be seen in ARCm and ARCv. This pattern is even valid and clear for the state-of-the -htth--rt ImageNet models. In brief, SAE is reflected by higher gradient consistency in ARCm, or greater a and smaller in ARCv. Similar visualization
99
100
101
102
103
104
105
106
107
from other PGD-like attacks, including PGD 2 MIM 3 and APGD 4 in Fig. 3 indicates the
109
possibility of generalization for all PGD-like attacks with only training samples from the BIM attack despite domain shift. We adopt SVM afterwards to retain explainability and simplicity.
3

---

FASSS
TAGOS
Figure 3: ARCm with adversarial examples created by PGD (left), MIM (middle), and APGD (right) attacks. The three rows correspond to ResNet- 18, ResNet-152, and SwinT-B-INIK, respectively. It is clear that PGD-like attacks qualitatively manifest similar SAE through ARCm.
111
Uniqueness of SAE to PGD-Like Attack. Whether SAE can be consistently triggered depends on whether the following conditions are simultancously true: (I) whether the input is adversarially perturbed by an iterative projected gradient update method; (II) whether the attack leverages first- order gradient of the model; (III) whether the Lp boundary types are the same for the two stages, i.e. attack and exploitation vectors: (IV) whether the loss functions for the two stages are the same; (V) whether the labels used (if any) for the two stages are relevant. Namely, only when the attack and exploitation vectors "match", SAE can be uniquely triggered as the exploitation vectors "continue" an attack, or they will "restart" an attack. Thus, in Fig. 1 Fig. and Fig. [3] all the conditions are true as they involve PGD-like attacks. We acknowledge the ARC being insensitive to non-PGD--like attacks (such C&W (12) is a limitation in practice. However, the unique SAE meanwhile shows possibility of inferring the attack details mentioned in the above conditions once triggered. SAE is
112
113
114
115
118
117
118
119
121
122
the trace of PGD-like attacks. Ablations for these five conditions are presented in Sec. 5
123
dappive Attack against ARC. Adaptive attacks can be designed against defense
13] or detec-
124
tion 14]. Likewise, they can be designed against ARC feature. To avoid SAE in ARCm, the adaptive attack must reach a point where the corresponding ARCm has mean value as small as that for benign examples. Intuitively, an adaptive attack has to simultaneously solve min, [[S,( + r)ll (Frobenius norm) alongside its original attack goal. however requires gradient of the Jacobians, namely at least T + 1 Hessian matrices, i.e., V2 (.) of size M x M to perform gradient descent. This is computationally prohibitive as in the typical ImageNet setting (i.e., ==33222x2224) a Hessian in float32 precision needs 84.4GiB memory, At this point, the cost of adaptive attack is much higher than computing ARC. We conclude that it is impractical to hide SAE from ARC at an acceptable cost without significant algorithm modification. The viable ways for attacker to avoid SAE is to use Non-PGD-like attacks or break the SAE uniqueness conditions. Being resistant to adaptive attacks while surviving our extremely limited problem setting is left for future study.
125
126
127
128
129
131
133
3
Attack Detection and Inferring Attack Details
135
136
Attack detection aims to identify the attempt to adversarially perturb an image even if it fails to change the prediction (but meanwhile left the trace) [ As demonstrated in the previous section, the SAE indicates the feasibility of attack detection specifically against PGD-like attacks.
137
138
Informed Attack Detection is to determine whether an arbitrary input म is adversarially perturbed, while the perturbation magnitude E is known. can be viewed a binary classification problem, where the input is ARCv of æ, and the output indicates "adversarially perturbed"", while 0 indicates "unperturbed". Thus, for a given E = 2*/255 where k E {1, 2, 3, 4). a corresponding Support Vector {0, 1} can be trained using some benign (s=0) samples and their adversarial counterparts (e=2* 255)). Even if the training data only involves the BIM attack, from visualization results, we expect generalization for other PGD-like attacks despite domain shift.
140
142
Machine (SVM) [13] classifier h
144
145
148
Uninformed Attack Detection is to determine whether an arbitrary input ø is adversarially perturbed, unknown. can be viewed as an ordinal regression [16]
147
while the perturbation magnitude problem, where the input is ARCv, and the output is the estimation of k. namely i: E 1, 2, 4). The corresponding estimate of & is é = 1 0}2²t /255, where ((- is the indicator function.
148
149
In practice it is undesirable to wait and react until the attack has succeeded.

---

MFCHrT S00M
GRONCE FCOOS
MEC
(fT) BIMIIARCE
(22) BIMIE Gaursian
(3) BIMIE Uniform
(64) FGSMIXber): ARCT
(E5) Guussian: ARCI
BIMI L2 ARCII
(ET) BIMII: L2
BIMILZ) LR
(19) BIMI ARCYC(a))
(110) BUMIL ARCIE?)
(1I) BIMIDE RJ: ARCU
(112) RIM[]: ARC[DER
(113) RIMIDER) RI, ARCTH
(f14) NESIXV f): ARC]]
(T15) SPSAJAT f1 ARCT)
Figure 4: Ablation on SAE uniqueness by adjusting exploitation vectors for ARC. Each subfigure of ARCm pair has two annotations: (1) attack and its settings, where empty brackets means default setting unless overriden: [Lp is Loo Loss is LCE: /(ss) iterative; // (can access) gradient Vf(-) (2) expoitation vector settings, e.g. "ARCIJ" with the default setting [L is Log Loss is LEE: Label is ö(-)]. The "c?" means random guess. This figure is supplementary to Tab.
150
Specifically, this is implemented as series of binary classifiers (SVM), where the k-th (k=0) classifier predicts whether the level of perturbation is greater equal k. i.e., whether k > k. Note, based on our visualization, the ARCv cluster of adversarial examples is moving away from that of benign examples as E (or k) increases. This means the ARCv of an adversarial example with > k will also cross the decision boundary of the k-th SVM h-).). Namely the SVM (+) can also tell whether i > k; and thus can be reused. Finally, the ordinal regression model can be expressed as the sum of prediction over the SVMs: k >>e (w). A perturbation is detected as long as i > O. Estimating & (or E) for & is similar to matching its ARCm position inside a much larger ARCm calculated starting from benign example. But, the estimate does not have to be precise, because the detection is already successful once any of the SVMs correctly raises an alert.
151
152
153
154
155
156
157
158
159
160
Although detector in practice knows completely nothing about potential attack including the attack type, evaluation of uninformed attack detection with known attack type is enough. Regarding the performance for uninformed attack detection given specific attack type of attack as a conditional performance, the expected performance in the wild can be calculated as the sum of conditional performance weighted by the prior probabilities that the corresponding attack happens.
161
162
163
164
165
Inferring Attack Details. Due to the SAE uniqueness in Sec. 2 once attack is detected, we can also predict that the attack: (I) is an iterative method performing projected gradient updates; (II) can access the first-order gradient of f()): (III) uses the same type of Lp bound as that in creation of exploitation vectors (Loc by default); (IV) uses the same function as that in creation of exploitation vectors (L.r(...) by default); (V) uses a ground-truth label which is relevant to the least-likely class
166
167
169
o(() used for exploitation vectors (in many cases é(σ) is exactly the ground-truth). In other words,
171
feasible post-pne post-procsssing defense is to correct prediction into the least-likely class é(Φ) upon detection.
172
Namely, the disadvantage of ARC being insensitive to non- non-PGD PGD-like attacks is meanwhile advantage
173
of being able to infer attack details of PGD-like attacks.
Experiments
174
175
In this section, we quantitatively verify the effectiveness of the ARC features in attack detection, and the performance of the post-processing defense under extremely limited setting. Unlike related works, the MNIST evaluation is omitted, as the corresponding conclusions may not hold [4] on 10, let alone ImageNet. We evaluate ResNet-18 [8] on CIFAR-10 77 ResNet- [8] and
177
179
SwinT-B-INIK [10 on ImageNet [9] with their official pre-trained weights (advantage of being
180
non-intrusive). Our code is implemented based on Py Torch [17). TorchAttacks [18] and Foolbox [19].
181
ARC Feature Parameter. For the BIM attack for exploitation vectors, we set step number T 6, and step size a 2/255 under the Lox bound with $/253. Note, the mean value of ARCm will tend to I with a larger T, making ARCv less separatable. We choose T to clearly visualize the value changes within ARCm, but this does not necessarily lead to the best performance.
183
5

---

Table I: Informed and Uninformed (the "E=?" column)) Attack Detection. All numbers are percentage with the "% sign omitted, except for MAE. Numbers greater than 50% are highlighted in bold font.
Dataset Model
Attack
2/255
4/255
8/255
c=
16/255
c=)
DR
FPR
Ace
Acc*
DR
FPR
Acc
Acc*
DR
FPR
Acc
Acc*
DR
FPR
Acc
Acc*
|MAE
DR
FPR
Acc
Aec
BIM
0.0
0.0
33.5
33.5
0.0
6.4
6.4
32.3
1.5
0.4
17.8
79.2
62.4
1.55
30.9
10.1
30.7
CIFAR-10
PGD
33.7
33.0
ResNe--18
MIM
0.3
0.0
30.4
30.4
0.0
0.0
6.5
6.5
37.5
1.5
0.4
22.5
84.5
1.1
0.0
67.4
1.50
33.6
1.5
9.3
32.4
APGD
0.0
0.0
29.3
29.3
0.0
0.0
5.1
5.1
36.9
1.5
0.2
20.7
78.8
1.1
0.0
55.8
1.53
31.5
1.5
8.7
28.0
AA
08
0.8
0.0
35.4
9.1
29.9
BIM
0.0
0.0
0.0
0.0
4.7
1.4
0.0
0.0
20.5
1.4
0.0
0.0
91.6
1.4
0.0
0.4
1.36
30.6
16
0.0
0.1
ImageNer
PGD
0.0
0.0
0.0
0.0
4.7
1.4
0.0
0.0
18.8
1.4
0.0
0.0
85.9
1.4
0.0
0.0
1/44
28.9
166
0.0
0.0
ResNet-152
APGD
0.0
0.0
0.0
0.0
2.0
1.4
0.0
0.0
11.3
1.4
0.0
0.0
61.7
1.4
0.0
0.4
1.59
19.7
166
0.0
0.1
AA
0.0
0.0
0.0
0.0
2.5
1.4
0.0
0.0
10.7
1.4
0.0
0.0
61.5
1.4
0.0
0.0
1.59
19.9
1.6
0.0
0.9
?
0.0
0.0
0.0
3.2
1,4
13.2
76.3
1.50
24.6
0.0
BIM
4.1
1.6
6.1
6.2
13.7
2,0
0.0
77.3
2,0
0.0
74.0
97.9
0.0
97.9
0.96
49.1
1.5
47.3
ImageNot
PGG)
3.9
1.6
23
3.1
16.4
2.0
0.0
10.9
72.7
2.0
0.0
68.8
98.4
0.2
0.0
98.4
1.01
48.6
2.0
0.5
45.9
SwinT-B-INIK
MIM
1.6
1.6
0.0
1.6
10.2
2.0
0.0
10.2
63.3
2.0
0.0
63.3
93.8
0.2
0.0
93.8
1.09
43.8
0.0
43.8
APGD
A-A
1.8
5.7
31.6
68.4
29.5
23.2
?
2.6
1.6
1.7
2.6
10.2
2.0
0.0
7.7
55.5
2.0
0.0
51.2
84.7
0.2
0.0
79.0
1.16
40.1
2.0
0.4
36.7
185
Training, To train SVMs h+)))) with RBF kernel, we randomly select 50 training samples from
186
CIFAR-10, 10, and perturb them using only BIM (1 with magniude e =
= 2/2555, 4/255, 8/255, 16/255, respectively. Then each of the four (.x(-) is trained with ARCv of the benign (s 0) samples and
187
perturbed (5 = 2* /255) samples. Likewise, for ImageNet we randomly select 50 training samples and train SVM in a similar setting separately for ResNet- and SwinT-B-INIK. The weight for benign sample can be adjusted for training in order to control False Positive Rate (FPR).
188
190
Testing. For CIFAR-10, all 10000 testing data and their perturhed versions with different are used to test our SVM. For ImageNet, we randomly choose 512 testing samples to test our SVM due to computation cost of Jacobian matrices. A wide range of adversarial attacks are involved, including (1) PGD-like attacks: include BIM M. PGD [2]. MIM (3) APGD
192
194
4 AutoAttack (AA) 44: (2) Non-PGD-like attacks: (2.1) other white-box attacks: FGSM O C&W 122 (we
196
use S E {0.5, 1.0, 2.0, 3.0) in Le case), FAB 20), FMN [2T]; (2.2) transferabillty--ass attacks:
DI-FGSM (22), TI-FGSM 23 (using Re as proxy): (2.3) score-based black-box methods:
198
NES [24], SPSA (25). Square [26). Existing attack detection methods seldom evaluate on many types
of attacks. AutoAttack is regarded Pikelike because APGD is its most significant component for attack success rate. Details of all attacks can be found in the supplementary code.
200
Metries We evaluate the SVMs using Detection Rate (DR, a.ka., True Positive Rate), well as False Positive Rate (FPR). For the post-processing defense method, we report the original classification accuracy for perturbed examples (denoted as "Acc") as well as accuracy after correction (denoted as "Acc*"), For ordinal regression, we also report Mean Average Error (MAE) for reference.
202
204
205
4.1
Informed and Uninformed Attack Detection for PGD-like Attacks
206
For each network, the corresponding SVMs are trained and evaluated as shown in Tab. [ Columns with a concrete E value are informed attack detection, while the "c=?" column is un- informed attack detection. As can be expected from visualization results, the ARCv clusters are gradually becoming separatable with e increas- ing, and hence the increase of DR. Notably, the large perturbations (i.e., e = 16/2 (55 are very hard to defend (27). but can be consistently and accurately detected across architectures. The ARC feature is especially effective for trin-Transformer. because this model transitions faster from being non- minerr to being linear than other architectures. Such characteristics are beneficial for ARC.
207
208
210
212
Figure 5: ROC of SVMs in Tab. 1 & Tab. 3
214
216
217
218
Upon detection of attack, our method corrects the prediction into the least-likely class as a post- processing defense. Success of such method depends on whether the attack is efficient to make ground-truth class least-likely, and whether the network is easy for the attack to make a class least- likely. From Tab. I both ResNet-18 18 and SwanTrannf have such property and lead to high
220

---

Table 2: Ablation on SAE uniquenss by varying attacks. The row (t1) is regarded as a baseline, and notation means "same as baseline" in order to ease comparison. SAE will only show consistent effectiveness across architectures when the four conditions in Sec. 2 are satisfied.
Altack
ARC
ResNet-18
w/ =7
RosNet-112
152 w =/
SWinT-B-INIH wir
=7
Nome
1/
Loss
Rer.
Label
MAF
FPR
Ape
MAF
13R
FPR
MAE
T3R
FPR
BIM
CF
Yes
Yes
CE
30.9
10.1
30.7
1.36
30.6
0.0
0.1
0.9
49.1
2.0
1.5
47.3
12
BIM
1.27
49.9
1.5
2.6
39.0
1.98
3.5
1.6
0.2
0.2
2.02
1.0
2.0
1.4
1.8
13
BIM
DLR
1.98
2.1
1.5
10.5
10.6
1.63
18.9
1.6
0.0
0.6
1.44
27.5
2.0
1.8
6.6
14
RGSM
No
1.96
3,4
1.5
50.3
29.5
1.63
18.6
1.6
84
6.8
144
27.1
2.0
44.9
12.4
CRW
C&W
0.0
0.0
2.3
2.03
2.0
FAB
FAB
10.5
2.5
2.03
2.0
17
FMN
FMI
1.99
1.4
1.5
8.8
8.6
2.02
2.1
1.5
0.0
0.0
2.03
0.8
2.0
0.0
0.0
18
DI-FGSM
DI-F
No
1.98
2.2
1.5
42.9
42.0
1.98
3.5
1.6
27.9
27.5
1.87
8.2
2.0
67.2
62.1
19
ILRGSM
ILFGSM
No
1.8
1.9
1.5
59.4
58.3
2.00
2.9
1.6
40.0
39.1
2.02
1.6
2.0
72.3
70.9
110
NES
No
4.7
38,6
39.4
1.68
3.1
26.3
2.02
2.0
50.6
49.4
11
SPSA
No
3.1
2000
2.7
2.0
50.6
02
Square
Square
No
1.99
1.6
85.7
84,3
2.02
2.1
1.6
68.6
67.4
1.84
10.2
2.0
77,9
70.1
113
Goussian
N/A
No
No
199
1.7
15
870
85.6
2.00
2.7
1.6
75.2
73.2
2000
311
2.0
W2.4
79.7
114
Uniform
N/A
No
No
199
18
1.5
86.5
85.0
197
4.1
1.6
73.6
70.9
184
10.2
2.0
818
73.2
classification accuracy after correction. For ResNet-152, the least-likely label is merely relevant (not identical) to the ground-truth due to network property during attack, and hence leads to effective detection but not correction (this will be explained in next subsection). In contrast, the correction
223
224
225
method performs best on Swin- Transformer, as it can restore classification accuracy from 0.4% to
36.7% even if both concrete type of PGD-like attack and are unknown ("Attack=?" row and "c=?"
227
column in Tab. D. assuming flat prior. By adjusting the weights assigned to benign examples, the decision boundary of SVMs can be moved and hence influence the FPR, as shown in in Fig. 5
229
4.2
Sequel Attack Effect as Unique Trace of PGD-like Attacks
230
The SAE is unique to PGD-like attacks, as requires five conditions listed in Sec. 2 hold for consistent effectiveness. To clarify this, we change the attack settings (quantitatively in Tab. 2 or the exploitation vector for ARCM (qualitatively on CIFARIO 10 in Fig. and then review these conditions:
231
232
233
(I). Iterative attack (Iter.). The single-step version of PGD, ie.. FGSM (t4, f4) does not effectively exploit the search space within the Lp bound, and hence will not easily trigger linearity and SAE. Only Swin Transformer slightly reacts against FGSM due to its own characteristics of being easy to be turned linear. Thus, SAE requires the attack to be iterative;
235
236
237
(II.
Gradient access (-)( Transferability-aaee based attacks (t8, 19) uses proxy model gradients to create adversarial examples, and hence could not trigger SAE. NES (t10, f14) and SPSA (t11, f15) can be seen as PGD using gradients estimated from only network logits, but can still not trigger SAE as it cannot efficiently trigger linearity. Neither does Square attack (112). Thus, SAE requires that the attacks use the target model gradient;
238
239
240
241
242
(III).
Same Lp bound. When the attack is BIM in La bound (t2. f6), SAE will no longer be triggered for ImageNet models, because the change of Lp influences perturbation search process. However, SAE is still triggered for CIFAR-10 possibly due to relatively low-dimensional search space. This means CIFAR-10 10 property does not necessarily generalize to ImageNet. When ARC is changed accordingly (f7, f8), the feature clusters are still separatable. Thus, SAE requires the same type of Lp bound for consistent effectiveness;
243
2M
245
246
247
248
(IV). Same loss. When the loss for the BIM attack is switched from LCB to DLR (t3, fll), the SAE is significantly reduced. However, if exploitation vectors are also created using DLR loss (f12, f13), SAE will triggered again. Thus, SAE requires a consistent loss function; (V). Relevant label. When the most-likely label ē(.) is used for exploitation vectors, it leads to the least significant SAE (19). Besides, even a random label (c?) leads to moderate SAE (f10), while the least-likely label c(ã) (which is ground-truth label in many cases) Icads to distinct SAE (fl). The most significant SAE correspond to ~(ã) = c(x). This means in order to maximize cross-entropy, large portion of output functions fn (-) has been triggered local linearity during attack. Thus, SAE requires a relevant label (if any) for exploitation vectors.
250
252
254
256
257
When the exploitation vectors are created using random noise (f2, f3). SAE is not triggered. Neither does random noise as attack trigger SAE (t13, t14, f5). Other PGD-llee attacks (t5, 16, 17) do not trigger SAE as well. A special case is targeted PGD-like attack, where the creation of exploitation vector needs to be use negative cross-entropy loss on the most-likely label to reach a similar level of effectiveness (this paper focuses the default untargeted attack to avoid complication).
258
260
7

---

Table 3: Comparison with existing methods that are compatible with our problem setting.
Method Menic
RIM
2235
(0:55
PGD
1253
11255
MIM
D23
APGD
(rESS [0055]]
1930
(4255]
AA
10155
RedNa111
N88 10
FRR
18
0.0
6.6
0.0
1.3
0.3
1.5
2.5
0.0
0.3
0.0
1.3
0.1
1.5
2.5
0.6
00
13
0.1
1.5
2.5
0.0
0.0
0.3
13
02
1.5
2.5
0.0
08
0.3
0.5
18
0.2
1.5
2.5
ARC
FPR:
128
0.0
0.0
00
0.0
12.3
1.3
79.2
1.1
309
1.5
00
0.0
0.0
00
33111
1.5
512
1.1
31.
1.5
00
0.0
00
0.0
32.5
13
525
1.1
336
1.5
000
0.0
0.0
0.0
36,9
1.5
788
1.1
31.
1.5
080
0.0
0.5
0.0
37.3
1.5
78.4
1.1
31.6
1.5
ImugeNa RonNet-153
2.9
41.6
29
9.1
26.
35.7
IPK
1.2
1.2
2.0
12
12
ARC
IPK
DR
0.0
0.0
20.5
1.4
91A
1.4
50.6
1.6
0.0
0.0
144
188
1.4
65.9
1.4
28.9
1.6
0.0
23
1.4
4.7
1.4
$12
1.4
233
1.5
0.0
0.0
2.0
1.4
113
14
61.7
144
19.7
1.
0.0
0.0
2.5
1.4
10.7
1.4
61.5
1.4
19.9
1.6
ImageNet JNI
NS8
4.5
0.6
42.4
1.3
47.5
413
49
418
1.2
47.1
12.3
29.3
12
4.5
16
31.
12
35.5
14
31.8
122
35.1
108
1.6
300
16
200
0.3
16
2.0
02
14
20
02
16
2.0
0.2
262
The non-PGD attacks, or PGD attacks do not meed all conditions cannot consistently trigger SAE across architectures because they provide a less "matching" starting point for exploitation vectors, and hence make the BIM for exploitation vectors "restart" an attack, where the network behaves non-linear again. Only when all the conditions different architectures, especially for ImageNet models. As for label correction, PGD-like attacks can effectively leak the ground-trute labels in the adversarial example, long the network allows the attack to easily reduce the corresponding logit value to lowest among all.
264
satisfied will SAE be consistently triggered across
286
268
269
In summary, SAE is the unique trace of PGD-like attacks. Although insensitive to non-PGID-in attacks for general attack detection, SAE is a specific signature [28], indicating the feasibility of correcting prediction upon detection of PGD-like attacks.
270
4.3
Comparison with Previous Attack Detection Methods
273
As discussed in Sec. 6 due to our extremely limited problem setting - (1) no auxiliary deep model;
(2) non-intrusive; (3) data- undemanding, the most relevant methods that do not lack of ImageNet
275
evaluation are 22 30, 31 32 33 But 30 31
32 33
still require a considerable amount of data
276
to build accurate (relatively) high-dimensional statistics. The remaining NSS (29) method craft 18- dimensional features from Natural Scene Statistics, which are fed into SVM for binary classification.
277
278
We adapt the trained SVMs in our ordinal regression framework as well, with reduced training set
279
size to 100 (50 benign + 50 BIM adversarial) for each SVM for fair comparison. All SVMs are tuned
280
to control FPR. The results and ROC curves for "z=?" task can be found Tab. 3 and Fig. 5 It noted that (1) SVM with the 18-D NSS feature may fail to generalize due to insufficient sampling (hence
281
282
the below-diagonal ROC); (2) NSS performs better for small E, but performance saturates with larger
283
because NSS does not incorporate any cue from network gradient behavior; (3) small e is difficult for ARC, but its performance soars with larger & towards 100%, which consistent and expected from our visualization; (4) SVM with ARCv can generalize against all PGD-like attacks, while NSS failed for MIM: (5) SVM with NSS may generalize against some non-PGe attacks (29] while
284
285
286
ARC could not due to SAE uniqueness; (6) SVM with the 2-D NSS feature ("Method 2" in (2 fails
286
to generalize. Thus, ARC achieves competitive performance consistently across different settings despite the extreme limits, because the ARC feature is low-dimensional, and incorporates cue from network gradient behavior. Apart from these, ARC also provides a new perspective to understanding attack and defense from model's gradient behavior, discussed in Sec.
290
Discussions and Justifications
292
Ordinal Regression. Intuitively, the uninformed attack detection can be formulated as standard regression to estimate a continuous & value. However, this introduces an undesired additional threshold dyper-parameter for deciding whether an input with e.g., 0.5 estimation is adversarial. Ordinal regression produces discrete & values and avoids such ambiguity and unnecessary parameter.
294
296
297
Training Set Size. Each of our SVMs has only 100 training data (i.e., 50 benign + 50 adversarial). The simple 2-D ARCv distribution (Fig. 2 can be reflected by few data points, which even allows an SVM to generalize with less than 100 data points (but may suffer from insufficient sampling with too few, e.g., 10+10 samples). contrast, the performance gain will be marginal starting from roughly
298
300
200 training samples, because the ARCv feature distribution is already well represented.
8

---

302
ing makes the model very linear around the data (35). As a new why adversarial training works, the networks are Combination with Adversarial Training, From our experiment and recent defenses [2] 34 27, its noted that (1) small perturba- tions are hard to detect, but easy to defend; while (2) large pertur- bations are hard to defend, but easy to detect. However, combining defense and our detection is not effective on ImageNet. As shown in Fig. 6, we compute ARCm based on regular ResNet-50 (from Py Torch (17 and adversarially trained ResNet-50 on ImageNet (from 34 ). Unlike the regular ResNet-50, adversarially trained one has much higher mean value in ARCm, and the resulting ARC vectors almost non-separatable. This means adversarial train-
304
305
306
308
310
312
Figure 6: ARCm from regu- lar (1#t row), and adversarially trained ResNet-50 (2nd row w/ e=4/255, 3rd row w/ 8-//255).
313
perspective
314
trained to generalize while being already very linear to the input, and thus it will be hard for attack to make the model behave even more linear to significantly manipulate the output.
315
318
Limitations. (1) The ARC Feature is only sensitive to the PGD-like attacks, and relies on the least-
318
likely assumption for effectiveness of prediction correction. But such selective sensitivity meanwhile leads to the uniqueness of SAE. (2) Jacobian computation is slow for ImageNet models because it requires 1000 iterations of backward pass. A single Jacobian of ResNet- 152 takes 161=0.55 seconds on Nvidia Titan Xp. Thus are unable evaluate our method on all ImageNet data with 2 GPUs.
320
322
Future Recommendations. (1) Include ImageNet evaluation, as CIFAR-10 property may not hold on ImageNet; (2) Check detector sensitivity w.r.f. attack algorithm parameter, as it may be significant.
323
Related Works
324
325
Adversarial Attack and Defense. Neural networks are vulnerable to attacks 36 6
To
exploit such vulnerability, attacks under different threat models are designed, including but not limited to white-box attacks 2123444 4), transfcrability-bat
327
attacks [37] 38 22 23). scsee
25 26) and retiision-basee black-box attacks 40) Different from these run-time attacks, backdoor attack 41) happens during the training. To counter the attacks, adversarial training 2 27 42 is the most promising defense make networks resistant the adversarial perturbations, but is meanwhile intrusive (i.e. requires retraining), and suffering from a notable generalization gap. Certified defense [43 and perturbation reverse engineering are also proposed [44). A defense may be invalidated by adaptive attacks [45] [3]. Our method to correct the prediction upon detection can be seen as a post-processing defense.
black-box attacks (39 24
329
330
331
332
333
334
335 Adversarial Example Detection [46. [4] aims to predict whether a given image is adversarial or not,
so that adversarial ones can be rejected. This can be achieved through adversarial training [47 (8). customized subnet [49] or customized loss [50]. but will be costly for ImageNet. Generative model-based detection methods check adversarial example reconstruction error [51] or probability density [52), but are data-demanding in order to learn accurate distributions. Auxiliary deep model [53]
337
339
54 for attack detection not only require large amount of data, but also susceptible to adaptive
341
attack [14. Dropout can be used for detection when combined with Bayesian uncertainty (55]. leverage (high-dimensional) features, which is the most compatible group of method to our problem setting, but most of them are still data- demanding for an accurate statistics. Whilst MNIST property may not hold on CIFAR-10 11,, let alone ImageNet, many related works lack the evaluation on ImageNet. Whilst detection difficulty varies with attack parameters, very large portion of related works have neglected the respective sensitivity analysis. Additionally, we point out conditions under which our method will be invalidated.
342
Feature statistics-based methods [31 30 29 32
33
343
344
345
346
347
Conclusions
348
349 In this paper, we design an Adversarial Response Characteristic (ARC) feature with an intuition that the model being attacked behaves more "linear" against adversarial examples than does to benign ones, which valid for PGD-like attacks in terms of attack detection and prediction correction. Our 352 method is weighte,, non-intrusive, data-undemanding and simple to interpret.
351

---

References
353
[1]
Alexey Kurakin, Ian Goodfellow, Samy Bengio, al. Adversarial examples in the physical world. The International Conference on Learning Representations Workshop Track, 2016.
355
[2] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant adversarial attacks. The International Conference on Learning Representations, 2018.
357
358
359
[3] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
360
361
362
[4] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning, 2020.
363
364
[5] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L.. Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
385
388
367
[6]
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. The International Conference on Learning Representations, 2015.
388
369
[7] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
370
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint 11X.v:151 2015.
371
372
[9] Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
373
374
375 [10] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2021.
376
378
[1I] Pauli Virtanen, Ralf Gommers. Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Eygeni Burovski, Pearu Peterson, Warren Weekesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis I Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020.
379
380
381
382
383
385
[12] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 39 57. IEEE, 2017.
387
[13] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 13:1633-1645, 2020.
389
[14] Nicholas Carlini and David Wagner. Adversarial Examples Are Not Easily Detected:
Bypassing Ten
Detection Methods, page 14. Association for Computing Machinery, New York, NY, USA, 2017.
381
[15] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E, Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
382
383
[16] Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output enn for age estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
396
[17] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 8035. 2019.
398
399
400
402
[18] Hoki Kim. Torchattacs:: A pytorch repository for adversarial attacks. arXiv preprint ar ar10:0000000 2020.
10

---

404
[19]
Jonas Rauber, Roland Zimmermann, Malthias Bethge, and Wieland Brendel. Foolbox native: Fast adversarial attacks benchmark the robustness of machine learning models in pytorch, tensorflow, and jax. Journal of Open Source Software, 5(53):2607, 2020.
406
407
[20] Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. International Conference on Machine Learning, pages 2196-2205. PMI.R, 2020.
408
409
[21] Maura Pintor, Fabio Roli, Wieland Brendel, and Battista Biggio. Fast minimum-norm adversarial attacks through adaptive norm constraints. In Advances in Neural Information Processing Systems, volume 34, pages 20052-20062, 2021.
410
412
[22] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai. Jianyu Wang, Zhou Ren, and Alan Yuille. Improving transferability of adversarial examples with input diversity. In Computer Vision and Pottern Recognition. TEEE, 2019.
414
[23] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariann attacks. In Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition, pages 4312 -2211, 2019.
416
417
418
[24] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box box adversarial attacks with limited queries and information. In ICML, pages 2137-2146. PMLR, 2018.
419
420
[25]
Jonathan Uesato, Brendan O' d'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In ICML., pages 5025-5034. PMLR, 2018.
421
422
[26] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. 2020.
[27] Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy (Dj) Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019.
426
427
[28]
Hossein Souri, Pirazh Khorramshahi, Chun Pong Lau, Micah Goldblum, and Rama Chellappa. Identifica- tion of altack-specific signatures in adversarial examples. CoRR, abs/21 10.06802, 2021.
428
429
[29] Anouar Kherchouche, Sid Ahmed Fezza, Wassim Hamidouche, and Olivier Déforges. Detection of adversarial examples in deep neural networks with natural scene statisties. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-7, 2020.
430
432
[30] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting ad- versarial examples. In Proceedings of the 36th International Conference Machine Learning, volume 97, pages 5498 5507, 09- Jun 2019.
434
435
[31] Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5775-5783, 2017.
436
437
[32] Jaajun Lu, Theerasit Issaranon, and David A. Forsyth. Safetynet: Detecting and rejecting adversarial examples robustly. CoRR, abs/1004.00113, 2017.
438
439
[33]
Shiqing Ma and Yingqi Liu. Nic: Detecting adversarial samples with neural network invariant checking. In Proceedings of the 26th network and distributed system security symposium (NDSS 2019), 2019.
440
441
[34] Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. httpp:://gtthu..co com www.///a/aaaa/oooooooooo www../stta.bbbbbss....o.o
442
443
[35] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. Adversarial training is a form of data-dependent operator regularization. In Advances in Neural Information Processing Systems, volume 33, pages 14973-14985, 2020.
445
446
[36] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Roh Fergus. Intriguing properties of neural networks. arXiv preprint arxiv: 1312,619, 2013.
447
448
[37] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.00 07277, 2016.
449
450
[38] Seyed- Mohsen Moosavi-Dezfoon Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adver- sarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1765 1173, 2017.
451
11

---

453
[39] Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi. and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15-26. 2017.
455
[40] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision- based adversarial attacks: Reliable attacks against black-box machine learning models. In 6th International Conference on Learning Represen- tations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
457
458
459
460
[41] Nicholas Carlini and Andreas Terzis, Poisoning and backdooring contrastive learning. In International Conference on Learning Representations, 2022.
461
462
[42]
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurlPS, 2020.
463
484
[43] Mathias Lecuyer. Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 JEEE Symposium on Security and Privacy (SP), pages 656-672. IEEE, 2019.
465
466
467
[44] Yifan Gong, Yuguang Yao, Yize Li, Yimeng Zhang, Xiaoming Liu, Xue Lin, and Sijia Liu. Reverse engincering of imperceptible adversarial image perturbations. In International Conference on Learning Representations, 2022.
468
469
470
[45] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pages 274-283. PMLR, 2018
471
472
473
[46] Ahmed Aldahdooh, Wassim Hamidouche, Sid Ahmed Fezza, and Olivier Déforges, Adversarial example detection for dnn models: review and experimental comparison. Artificial Intelligence Review, Jan 2022.
475
[47] Xuwang Yin, Soheil Kolouri, and Gustavo K Rohde. Gat: Generative adversarial training for adversarial example detection and robust classification. In International Conference on Learning Representations, 2020.
476
477
478
[48] Xuwang Yin. Soheil Kolouri, and Gustavo K. Rohde. Divide-and-conquer adversarial detection. CoRR, 1b4/1905.11455, 2019.
479
480
[49] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint 17Xiv:1702.04267, 2017.
482
[50] Tianyu Pang. Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial examples. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, page 1584-4594, Red Hook, NY, USA, 2018. Curran Associates Inc.
484
[51] Dongyu Meng and Hao Chen, Magnet: A two-pronged defense against adversarial examples. In Proceed- ings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 117, page 135-147, New York, NY, USA, 2017, Association for Computing Machinery.
486
487
[22] Yang Song. Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In International Conference on Learning Representations, 2018.
489
490
491
[53] Gaurav Kumar Nayak, Ruchit Rawal, and Anirban Chakraborty. Dad: free adversarial defense at test time. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3562-3571, 2022.
492
493
494
[54] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Jun Zhu, and Xiaolin Hu. Defense against adversarial attacks using high-level representation guided denoiser. CoRR, ahs/1712.02976, 2017.
495
496
[55] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1103.00410, 2017.
497
498
[56] Shengyuan Hu, Tao Yu, Chuan Guo, Wei-Lun Chao, and Kilian Q Weinberger. A new defense against adversarial images: Turning a weakness into a strength. Advances in Neural Information Processing Systems, 32, 2019.
499
500
501
[57] Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Newral Information Processing Systems, volume 33, pages 1008-16659, Curran Associates, Inc., 2020.
502
12

---

504
[58] Soorya Gopalakrishnan, Zhinus Marzi, Upamanyu Madhow, and Ramtin Pedarsani. Combating adver- sarial attacks using sparse representations. Sixth International Conference Learning Representations, Workshop Track, 2018.
506
[59]
Ambar Pal and Rene Vidal. A game theoretic analysis of additive adversarial attacks and defenses. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin. editors, Advances in Neural Information Processing Systems, volume 33, pages 1345-1355. Curran Associates, Inc., 2020.
508
509
600] Peter Bartlett, Sebastien Bubeck, and Yeshwanth Cherapanamjeri. Adversarial examples multi- multi-layer random relu networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 9241 9252. Curran Associates, Inc., 2021.
511
512
13

---

514
A
Additional Discussions
515
A.1
Summary of Pros & Cons of the Proposed Method
518
Pros:
517
Relies on strong assumptions and hence is specifically effective for PGD-like attacks. Namely, the unique trace of PGD-like attacks can be used in specific (instead of generic) defense scenarios with knowledge about the attacker, or forensics scenarios to tell whether adversarial example is created by PGD-like methods by identifying the unique trace.
518
519
521
Can infer other attack algorithm details such as loss function and the ground-truth labels, while the other attack detection methods cannot do the same.
522
528
Easy and straightforward interpret for human, since the meaning of the ARC features is clearly defined, and the feature dimensionality is low.
524
525
Light- weighted in terms of algorithm components, No any additional deep neural networks is required.
non-intrusive. Does not require any change in neural network architecture parameters. The proposed method analyzes the Jacobian matrices calculated from the neural network of interest.
528
Data-undemanding. Does not require a large number of training data. We use merely 50 training samples in our experiments.
531
532
The stronger the attack is, the stronger the trace (and hence the higher detection rate). Previous methods compatible to our extremely-limite setting do not have such property and may even perform worse with large perturbations in some cases (See Table 3).
533
534
535
Reveals new perspective to understand why Adversarial Training works. (See "Combina- tion with Adversarial Training" in Section 5).
536
537
Cons:
Relies strong assumptions (See "Uniqueness of SAE to PGD-Like Attack" in Section 2), and hence not effective under non-PGD scenarios since assumptions are broken. Ablation studies are carefully carried out in Section 4.2 to examine and justify these assumptions.
539
541
Suffers from high time complexity due to Jacobian matrix calculation. In practice this is reflected by time consumption of calculation of the ARC feature (See "Limitations" in Section 5). Experiments on ImageNet are extremely slow and hence we are unable to evaluate the method on all ImageNet data.
542
544
545
Performs worse than previous NSS method against small perturbations (i.e., 5 2/255 or == 44/255). (But significantly better against large perturbations).
546
547
Incompatible with Adversarial Training. (But meanwhile provides a new perspective to understand why adversarial training works. See "Combination with Adversarial Training" in Section 5).
548
519
550
A.2
Iterations of PGD-like Attacks
is known that the number of iterations (fixed at 100 in experiments) also impacts the attack strength besides perturbation magnitude s. As increasing number of iterations will also lead to a more linear response from the model given an fixed and appropriate e and achieve SAE similarly, we stick to one controlled variable s for simplicity.
552
553
554
On the contrary, reducing the number of iterations of a PGD-like attack will also lead to small perturbations that are hard to detect (as demonstrated in Section 4), and hence increase the possibility that the attack will not trigger clear SAE and hence bypass the proposed detection method. As an extreme case, FGSM, namely the single-step version of PGD does not effectively trigger SAE (as discussed in Section 4.2).
556
558
559
560
The related works usually fix at a single set of attack parameters, and hence miss the observation that smaller perturbations are harder to detect.
14

---

582
A.3
Motivation of Extremely Limited Setting, including Limited Data
563
An extremely limited problem setting (Paragraph in Section 1) makes the proposed method flexible and applicable in wider range of defense and forensics scenarios compared to existing methods. Namely, a method can be used in more flexible scenarios it requires less from the adopter.
584
565
566
Limited number of data samples. Data-demanding methods is only applicable for models using publicly available datasets, or is only applicable by the first-party who trained the neural network. This limits the use cases of these methods. In contrast, we do not assume collecting a large amount of data is easy for potential adopters of the proposed method. Due the low demand on data, the proposed method enables a wider range of defense or forensics scenarios, especially when there is no access to the whole training dataset. For instance, the "Third-party Attack Detection Forensies" and Attack Detection for Federated Learning" scenarios.
567
568
589
570
571
572
thirt-party Attack Detection (identify whether the model is attacked) Forensics (identify attack type and infer the attack detail). Being data-undemanding means the proposed method can be applied to any pre-trained neural network randomly downloaded from the internet, or purchased from an commercial entity. For pre-trained neural networks using proprietary training datasets with commercial secret or ethic/privacy concerns (such commercial face datasets and CT scans from patients). the proposed method is still valid as long as there are are few training samples for reference, or possible to request a few reference training samples.
574
575
576
577
578
579
580
581
Attack Detection for Federated Learning. In federated learning, raw training data (such as face images) is forbidden to be transmitted to the central server. And hence even the neural network trainer cannot access the full training dataset (will violate user privacy), and is impossible to use any data-demanding methods to detect attack against a trained model (e, g., face recognition model). In contrast, the proposed method is still valid in this scenario as long as a few training samples can be collected from several volunteers for reference.
582
583
585
587
Many models deployed in production are unaware of adversarial attack. Re-training and replacing these models will induce cost, and will even introduce the risk of reducing benign example performance.
No change to network architecture or weights.
588
589
580
No auxiliary deep networks. Since a large amount of data is assumed to be not easy to obtain due to commercial or ethic reasons, training auxiliary deep networks are not always feasible. Pre-trained auxiliary deep networks are not always available for any classification task.
581
592
593
A.4
More on Adaptive Attack
594
According to 13. some similar attack detection methods are broken by adaptive attacks. Here we 595
discuss more about the existing adaptive attacks and report the quantitative experimental results. We also further elaborate on the adaptive attack mentioned in Section 2.
587
Logit Matching, (from Section 5.2 "The Odds are Odd" of (13) Instead of maximizing the default entropy loss, we switch to minimize the MSE loss between the clean logits from another class and that of the adversarial example. We conduct experiment with all testing data from CIFAR-10. and
598
599
600
128 random testing samples from ImageNet (due to limited time frame of rebuttal). The experimental
601
results can be found in the following table. Note, switching loss function to MSE loss (Logit Matching) breaks our assumption (IV). However, the attack still triggers SAE through the least-likely
602
603
class, and hence our method is still effective, but is (expectedly) weaker compared to the BIM with the original cross-entropy loss.
Datael
Attack
TI
Acc
DX
199E
11%
TI
As
CIFAR-1D
BIM (Logat Marching)
0.0
00
906
80.
|
00
08
632
55.2
23.8
1.
463
385
46.0
1.1
35.0
2022
1288
1.5
57.1
46.
ResNes-88
InsaeNes
BIM LLggi Marching
60
461
46
70
112
70
914
303
176
FraggNaa
misdeggi Marzinng
43
170
383
10
23
133
131
SwinT B-IIT.BINNIK
Table 4: Results of Logit Matching as adaptive attack against our method.
605
Interpolation with Binary Search. (from Section 5.13 "Turning Weakness into a Strength" of
[3) This methods find interpolated adversarial examples that are close to the decision boundary
15

---

607
with binary search. We conduct experiment with all testing data from CIFAR-10, and 128 random testing samples from ImageNet (due to limited time frame of rebuttal). The experimental results can 608
be found in the following table. Compared to the baseline results, the results show that our method is still effective against the adversarial examples close to the decision boundary.
610
Madl
Attack
DR
2/955
Are
Aue
12
PPR
4/550
Auc
DR
FPR
5/155
DR
PR
16/255
Ane
UR
PPR
Ape
BIM BlMilaoopelatto)
0.
00
65.7
65.
466
280
13
21.
280
50
R5
BIMT liegopettoooo 000 188
188
35
1250
11
08
08
50
08
ReaNe-
ImigaNii
MMMrrrieeeenn T6
T6
RS
B3
1o
18
2u
or
9o
20
Swait-B- INIK
Table 5: Results of Interpolation with Binary Search as adaptive attack against our method.
611
Adaptive Attack discussed in Section 2. To avoid triggering SAE, the goal of the PGD attack can
612
include an additional term to minimize [= + lll... Namely, the corresponding adaptive attack is:
arg max LCE + r, c(x)) - ||.. + +
= arg max LCr + (, (x))) - 12 Σ5
(1:))) 1
1 3
T41 T+1
arg max max
LCE( + r, (x))
1
Σ
cos cos fn- (x + 8.), V.x +r + 5,)1211/2
613
To solve this adaptive attack problem, the straightforward solution is to conduct Z-step PGD updates with the modified loss function. Each step includes but is not limited to these computations: (1) T1 Jacobian matrices to calculate n² and V fz (.); (2) T' + 1 Hessian matrices to calculate f2 (-). Let sb and Th be the time consumption for Jacobian and Hessian matrices respectively. Then the time consumption of the Z steps of optimization in total is greater than Z(T + 1) + v)H).
615
616
617
618
For reference, for Nvidia Titan Xp GPU and CIFAR- CIFAR-10/ResSV 10/ReeNet- 18, the Φb = 0.187 = 0.012 seconds, and 6 = 20.959 + 0.679 seconds (Python code for this benchmark can be found Appendix). we use Z = 100 steps of PGD attack, and T = 6 for calculating ARC, each adversarial example of CIFAR-10 image takes more than Z(T + ( + v/)) R 14802 seconds (i.e., 4.1 hours).
619
621
622 Note, we acknowledge that other alternative adaptive attack designs are possible, but as long as the alternative design involves optimizing any loss term calculated from gradients, second-order gradients
623
624
(Hessian) will be required to finish the optimization process, which again makes the alternative attack
625
computationally prohibitive.
A.5
More Related Works
627
We discuss the related works in more details, as an extension to Section 6.
628
Similar Defenses.
"The Odds are Odd" 30 is attack detection method based on feature statistical test. This method is categorized in Section 6 as feature statistics- based methods. In particular, it detects adversarial examples based on the difference between the logits of clean image and image with random noise. This method assumes that a random noise may break the adversarial perturbation and hence lead to notable changes in the logits, and is is capable of correcting test time predictions. Meanwhile, it can be broken by adaptive attack to match the logits with image from another example [ Similarly, our method can be seen as statistical test for gradient consistency as reflected by ARC feature. Our method is motivated by the assumption that neural networks will manifest "local linearity" with respect to adversarial examples, which will not happen for benign examples. Meanwhile the SAE is consistent across different architectures, and the corresponding 2-D ARCv feature shows very simple cluster structure for both benign and adversarial examples. The adaptive attack against (30] can merely slightly reduce the effectiveness of our attack, as shown in the additional adaptive attack experiments in this Appendix.
830
832
638
634
636
637
638
639
640
641
612
643
"Turning a Weakness into a Strength" (56) is an attack detection method which is concep- tually similar to [30]. This method involves two criterion for detection: (1) low density
16

---

645
of adversarial perturbations random perturbations applied to natural images should not lead changes in the predicted label. The input will be rejected if the change in predicted probability vector is significant after adding Gaussian noise. (2) close proximity to the decision boundary- - this leads to method that rejects an input if it requires too many steps to successfully perturb with an iterative attack algorithm. Hence, this method can be seen as an detector with two-dimensional manually crafted feature. This method can be broken by adaptive attack [33 that searches for interpolation between the benign and adversarial example. Similarly, our method leverages BIM, an iterative attack to calculate the ARC feature. However, differently, around the input, in order to calculate the extent of "local linearity" around the point as the ARC feature, while [56] leverages iterative attack to count the number of required steps. The ARC feature shows clear difference between benign and adversarial examples, and hence does not need to combine with other manually crafted feature. [56] points that
648
648
649
650
652
method use the iterative attack to explore the local area
654
656
658
solely using one criterion is insufficient, because the criterion (1) may be easily bypassed.
The adaptive attack against [56] can merely slightly reduce the effectiveness of our attack, as shown in the additional adaptive attack experiments in this Appendix.
680
661
Local Linearity, Local linearity is an important characteristies for the community to understand the adversarial attack as well as design defense methods.
662
663
FGSM (6) is designed based on the intuition that neural networks are vulnerable because their "local linear" property has been triggered by the attack. This is the first work that propose the concept of "local linearity" about adversarial attack. Many follow-up works about "local linearity" are adversarial training methods.
685
In LLS [27] (adversarial training), regularizer is proposed that encourages the loss to behave linearly in the vicinity of the training data, thereby penalizing gradient obfuscation while encouraging robustness. This is relevant to our interpretation on adversarial training in Section 5,
668
670
671
In GradAlign 57 (adversarial training), it is noted that the network being highly non-linear locally is the main reason why FGSM training fails.
672
673
Sparsifying front end [58] points out that a "locally linear" model can be used to develop theoretical foundation for crafting attacks and defenses.
674
675
In [59]. it is proved that the Fast Gradient Method attack and a Randomized Smoothing defense form a Nash Equilibrium, under a locally linear decision boundary model for the underlying binary classifier.
678
677
678
[60) shows that local linearity arises naturally at initialization.
679
A.6
Python Code for Evaluating Time Consumption of Jacobian / Hessian Calculation
680
The python code for measuring the time consumption for Jacobian and Hessian matrices calculation shown below. The code is based on CIFAR-10 10 settings with M and the neural network used is ResNet-18, 8. For reference, the result on Nvidia Titan Xp GPU is
681
32 x 32 and N 10.
682
683
0.187 + 0.012 seconds for Jacobian, and 20.90 20.959 + 0.679 seconds for Hessian.
684 Note, for the ImageNer/ResNet-1522 case, the Jacobian and Hessian calculation cost is much higher.
import time, torch as th, torchvision as V, numpy as np device 'cuda'
resnet18 wtw.meel....eeetteee.e rew.et18 to (device) # standard resnet18 resnet18.eval
resnet18. fc th mn ww..nn..iin Lonear (512, 10) to (device) # fit for 10 classes X th. rand (1, 3, 32, 32). to # random input # compute jacobian
time_start = time. time()
th. autograd. functional, jacobian x)
time_emm time. time
17

---

('A Jacobian takes:?, time_end compute hesstan
time start, 'seconds') ))
time_stare = time. time()
H th. autograd. functional. hessian (lambda x: resnet18(x) [o, o], X)
time_end time
print('A Hessian takes:', time_end time_start, 'seconds')
18

---

Checklist
685
686
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or []/A] . You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
687
688
690
Did you include the license to the code and datasets? [Yes] See Section ??.
691
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
694
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
695
696
697
1. For all authors...
698
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Contributions summarized at the end of Section T
689
700
(b) Did you describe the limitations of your work? [Yes] Limitations are summarized at the end of Section [
701
702
(c) Did you discuss any potential negative societal impacts of your work? [No] Attack detection is expected to build safer and more secure applications. Positive societal impacts are expected.
703
705
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
706
707
2. If you are including theoretical results...
708
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
711
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] Code is included in supplementary material.
712
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All training details are included in Section 4
715
(c) Did you report error bars (e.g.. with respect to the random seed after running experi-
717
ments multiple times)? [N/A] SVM converges to a reproducible result.
718
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We mentioned the computer resource at the end of Section 5 Our experiments are carried out with two Nvidia Titan Xp experiments.
719
720
721
722
4. If you are using existing assets code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Dataset papers cited. Compared methods are cited.
724
725
(b) Did you mention the license of the assets? [N/A] The CIFAR-10 dataset webpage hhtpps://ww cs. toronto. wwtp/ddii//eee/eeeeeeeeaeaaaa ciiar .html does not specify license. Im- ageNet dataset license can be found at https: : /www. image- org/ downloal php The pretrained models available for public download. including ResNet- 152 from PyTorch, and SwinT-B-INIK are not specified with a license. The authors of code of compared method do not specify their license.
726
727
728
730
731
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
732
There is no new assets in this paper.
733
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
19

---

735
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
736
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
739
740
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
742
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
20


Reviews:
[{"review_id": "zLbl2g-jOPx", "reviewer": "Reviewer_nyro", "summary": "This paper proposes a new and interesting method to detect PGD-like adversarial perturbations. The insight is to leverage the fact that PGD-like adversarial attacks will trigger local linearity of the input sample. To quantitatively measure the local linearity, the authors propose to use the Sequel Attack Effect (SAE) to continue the attack with BIM and then measure linearity with Adversarial Response Characteristics (ARC). SVM-based models are then trained and used for the detection of informed/uninformed attack detections.", "questions": "- I understand the main contribution here is for detection with limited amount of training data. I wonder how well the performance of the proposed method would be given sufficient training data compared to other SOTA detection methods?\n- A minor question: why use T=48 in line 100? It would be better to give a brief explanation for special numbers.", "limitations": "Limitations and societal impact are comprehensively discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The proposed method is very interesting and novel, providing great contributions to understanding and detecting PGD-like adversarial attacks and defense.\n- The investigated topic is important and useful.\n- Extensive empirical results on CIFAR-10 and ImageNet well illustrate the effectiveness of the proposed method and support the claims.\n- The paper is well-written and easy to follow. Provided figures clearly show the insights of the method.\n\nWeakness\n- The method is only limited to PGD-like adversarial perturbations, unable to other popular attacks like C&W as discussed in the paper.\n- The importance of limited training data for adversarial detection should be better motivated in introduction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ks3TllR9f2U", "reviewer": "Reviewer_XmZm", "summary": "This paper proposed a method to trace the strong adversarial attacks (specifically PGD). The insight is PGD attack is likely to leave a trace and trigger the local linearity of the network. They then introduced the ARC to capture the SAE, resulting in a detector that can identify the informed attacks and uninformed attacks. Experiments including ImageNet showed the effectiveness.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrength:\n1. It studies very extremely limited settings for the defender, which is novel and interesting. \n2. The paper presents a host of visualization to support the intuitions and conclusions. \n\nComment:\n1. My major concern is that ARC is not effective under a lot of scenarios (including FGSM, small $\\epsilon$, and transfer-based attacks ). It is also possible for PGD attacks to bypass the detector by reducing the number of steps. It cannot work with a robust model (adv training). I am not sure if ARC is a valid trace method. \n\nAfter rebuttal, I am willing to improve the score to 5, but my concern about the method still exists. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N6w3yvLDbBY", "reviewer": "Reviewer_vEc7", "summary": "This paper proposes a detection defense against adversarial examples generated by untargeted PGD-like attacks. This defense relies on the assumption that such adversarial examples trigger distinguishable patterns of linearity along a local trace defined by the BIM attack, which maximizes the cross-entropy loss that regards the least-likely label as the ground truth. The linearity pattern is characterized by the ARC feature, a vector reduced from the jacobian matrices along the BIM trace. Extensive qualitative and quantitive results on CIFAR10 and ImageNet (ResNet and SwinT) demonstrate the distinguishability of the ARC feature over benign inputs and adversarial examples generated by PGD-like attacks. This characterization is then used to detect PGD-like attacks and infer their parameters. The overall defense is effective against non-adaptive (unaware of the defense) PGD-like attacks, and outperforms another detection defense that does not require too much data to collect statistics. Finally, some interesting observations are discussed, such as linearity enforced by adversarial training.", "questions": "My current score is based on the strengths of the proposed characterization method and the expected fix of minor weaknesses noted above. I am willing to raise my score if the following major concerns are adequately clarified or justified.\n* [Quality-Weakness-1] What if the least-likely label is not the ground truth label?\n* [Quality-Weakness-2] Discuss if the defense is indeed lightweight compared with previous defenses requiring auxiliary models.\n* [Quality-Weakness-3.1] Discuss the computational cost of the proposed adaptive attack on CIFAR-10.\n* [Quality-Weakness-3.2] Discuss the two adaptive attacks mentioned above.\n\nI am open to decrease the significance of the last question if the meta-reviewer deemed it as beyond scope or outweighed by the strengths of the proposed characterization method in a non-adversarial setting.", "limitations": "Since this paper claims to propose a defense, its main limitation is the lack of a sufficient discussion of adaptive attacks. Although the authors have provided some discussion at L123-134, I find it hard to be convinced that a more thorough evaluation is not necessary. It is strongly recommended to at least evaluate published adaptive attacks (that break previous similar defenses) on the proposed defense.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Before heading to the detailed comments, I would like to note *neutrally* that the proposed defense is similar in spirit to some previous detection defenses that rely on the adversarial example's higher robustness against benign noise [1] and adversarial perturbation [2], which were later broken by [3]. *This similarity leads to both pros and cons, detailed below.*\n\n[1] The Odds are Odd: A Statistical Test for Detecting Adversarial Examples. Roth et al. ICML 2019.\n\n[2] A New Defense Against Adversarial Images: Turning a Weakness into a Strength. Hu et al. NeurIPS 2019.\n\n[3] On Adaptive Attacks to Adversarial Example Defenses. Tramèr et al. NeurIPS 2020.\n\n### Originality\n\n**Strengths (major)**\n* The proposed characterization of adversarial examples is novel and only requires a tiny set of training data, which seems to be less discussed in the literature.\n* The proposed ARC feature is novel and explicitly characterizes a well-defined property (local linearity) of adversarial examples. I believe this is more sophisticated than previously explored properties such as robustness to benign noise [1] and adversarial perturbation [2].\n* The idea of using BIM attacks to characterize the trace of untargeted PGD-like attacks is insightful. The proposed defense can detect perturbed-but-unsuccessful inputs, which seemed not to be covered by previous defenses [1, 2].\n\n**Weaknesses (minor)**\n* **Discuss similar defenses.** It is suggested to discuss [1, 2] and potentially other similar defenses with more details on the underlying properties. For example, how the property explored in this work is different and superior to those explored by previous defenses.\n* **Discuss local linearity.** Since this work relies on the local linearity of adversarial examples, it is suggested to include more discussion about newer work in this direction to solidify the underlying assumption.\n\n### Quality\n\n**Strengths (major)**\n* The proposed characterization of adversarial examples is technically sound and well supported by experimental results on CIFAR-10 (ResNet) and ImageNet (ResNet and SwinT). It can effectively distinguish between benign inputs and adversarial examples generated by PGD-like attacks with different parameters.\n* The ablation study demonstrates the defense's non-sensitivity to non-PGD-like attacks, but the pros and cons are adequately justified.\n\n**Weaknesses (major)**\n\n* **Some assumptions are too strong.** This paper made some strong assumptions that may not always hold. For example, at L81 the authors assume that the input's ground-truth label is simply the least-likely one. While this is likely to hold for vanilla attacks, a slightly smarter attacker could simply make the ground-truth label anywhere between the most-likely and least-likely label to break this assumption. It is unclear how the original characterization performs in this case. For example, is it possible that the linearity goes down at the first few steps (given an incorrect guess of ground truth) before heading up in those cases?\n\n* **The proposed defense may not be lightweight**. Since the proposed defense requires several steps of BIM attacks at the inference stage to obtain the input's ARC feature, the defended model's inference of each input would now include overheads (both efficiency and memory) of several backward passes. While other defenses require auxiliary models (thus argued as not lightweight in this paper), those defenses do not involve backward passes and thus may be more efficient than the proposed defense. This weakness is partially supported by the authors' inability to evaluate all ImageNet data. It is suggested to include some empirical results to resolve or clarify this weakness.\n\n* **Insufficient discussion of adversarial robustness.** Since this paper claims to propose a defense against adversarial examples, I find it hard to be convinced that the proposed defense would come with much robustness against adaptive attacks. I understand that the authors have discussed the hardness of adaptive attacks to some extent and left them for future study (L123-134), but the current discussion is rather limited, even from the perspective of *published* adaptive attacks on similar defenses. *I am outlining my comments below but will be open on this point if the meta-reviewer deemed the following discussion as beyond the scope or outweighed by the strengths of the proposed characterization approach (in a non-adversarial setting).*\n  * At L123-134, the authors discussed an adaptive attack that is computationally prohibitive on ImageNet. However, it is unclear if the same claim holds for the smaller CIFAR-10, which is also a dataset evaluated in this paper. **Please discuss the computational cost of this adaptive attack on CIFAR-10 and if it is still prohibitive.**\n  * If I understand correctly, the adaptive attack must reach a point whose BIM-trace is equally non-linear (or has the same linearity pattern) as that of a benign input. **However, the current defense cannot prevent the existence of such points, so it is still possible to find them.** This is similar to previously broken defenses that expect adversarial inputs to have unique patterns in terms of robustness to benign noise [1] and adversarial perturbation [2]. Therefore, I believe discussing their *existing* adaptive attacks from [3] would significantly strengthen this paper. In particular, it is suggested to discuss the following two *published* adaptive attacks.\n    * Logit matching (Section 5.2 of [3]). If the PGD attacker now aims to reach a point whose logit is sufficiently close to an unperturbed image from a different class, can the proposed characterization still distinguish between benign inputs and such adversarial examples?\n    * Interpolation between the adversarial and benign examples (Section 5.13 of [3]). If the attacker moves the adversarial example generated by PGD towards the original benign input by interpolation, can the proposed characterization still distinguish between the original benign input and the adversarial example moved towards it?\n\n\n### Clarity\n\n**Strengths (major)**\n* The overall presentation is good. I appreciate the clear demonstration in Figure 1.\n* The summarized conditions in Section 4.2 is good.\n\n**Weaknesses (minor)**\n* The setting discussed at L24-26 comes out without any context, making it hard to understand why it is important and hard to achieve. It is suggested to motivate this setting with some related work and emphasize it more throughout the paper.\n* Similarly, at L59 I can see that the setting is extremely limited, but what are the strong \"cues\" and why are they hard to solve?\n\n### Significance\n\nI appreciate the great effort in characterizing the trace of PGD-like attacks and the extensive experiments; this paper might inspire some work in that direction. However, my biggest concern about this work is its robustness as a defense against adversarial examples, given my experience of the commonly acknowledged importance of evaluating adaptive attacks in the adversarial example defense literature. That being said, I am open to discussion on this point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "J_Khri3h609", "reviewer": "Reviewer_YXrD", "summary": "The authors propose a new detection method to detect PGD-like attacks. The detection method is based on the analysis of local linearity between benign samples and adversarial samples. The authors also introduce ARC and SAE to measure the local linearity. The experimental results indicate that this method can successfully detect PGD-like attacks on various datasets and model architectures.", "questions": "1. The authors claim that this method is only sensitive to PGD-like attacks. Is it because that only PGD-like attacks can trigger the local linearity? If a PGD-like attack does not trigger it, can the detection still sensitive to this attack?\n2. In Figure 5, what is the reason that NSS is a better detection method on ResNet-152?", "limitations": "The authors have addressed the limitations. I appreciate it.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think using gradients to detect AEs is interesting, and there are few works focusing on this point. However, there are some weaknesses.\n\n1. The design of adaptive attack is not convincing. To minimize the Fro-norm of S_*(x+r), the attacker only needs to adopt an additional loss term to minimize the difference between the gradients of AEs and clean data.\n2. There are five requirements for SAE to be consistently triggered. It seems that breaking requirements (III), (IV) and (V) are very easy. The threat model is too strict for me. The victim needs to have perfect knowledge of the attacker. \n3. As a detection method, its premier requirement is fast and cannot influence the performance (e.g., the predicting speed and accuracy) of the classifier. This detection is too slow to use in practice. \n4. The DR in Table 2 is very low, even for the baseline (t1).\n5. For ImageNet, the perturbation size is usually very small, i.e., 4./255 or less. So, the results in Table 3 indicate that this detection method cannot efficiently detect AEs for ImageNet.\n6. The baseline method is only NSS, which is not enough. Although the proposed method is data-undemanding, I think a victim can freely gather as much as clean data to train a detector. It should not be a problem to compare your method with other previous works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject