Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples
Authors: Maura Pintor
Luca Demetrio
Angelo Sotgiu
Giovanni Manca
Ambra Demontis
Nicholas Carlini
Battista Biggio
Fabio Roli
Venue: NeurIPS 2021 Submitted (2021)
Keywords: adversarial machine learning
machine learning

Abstract: Evaluating robustness of machine-learning models to adversarial examples is a challenging problem. Many defenses have been shown to provide a false sense of security by causing gradient-based attacks to fail, and they have been broken under more rigorous evaluations. Although guidelines and best practices have been suggested to improve current adversarial robustness evaluations, the lack of automatic testing and debugging tools makes it difficult to apply these recommendations in a systematic manner. In this work, we overcome these limitations by (i) defining a set of quantitative indicators which unveil common failures in the optimization of gradient-based attacks, and (ii) proposing specific mitigation strategies within a systematic evaluation protocol. Our extensive experimental analysis shows that the proposed indicators of failure can be used to visualize, debug and improve current adversarial robustness evaluations, providing a first concrete step towards automatizing and systematizing current adversarial robustness evaluations.

Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples
Anonymous Author(s)
Affiliation Address
email
Abstract
Evaluating robustness of machine-learning models to adversarial examples is a challenging problem. Many defenses have been shown provide a false sense of security by causing gradient-based attacks to fail, and they have been broken under more rigorous evaluations. Although guidelines and best practices have been sug- gested to improve current adversarial robustness evaluations, the lack of automatic testing and debugging tools makes it difficult apply these recommendations in a systematic manner. In this work, we overcome these limitations by (i) defining set of quantitative indicators which unveil common failures in the optimization of gradient-based attacks, and (ii) proposing specific mitigation strategies within a systematic evaluation protocol. Our extensive experimental analysis shows that the proposed indicators of failure can be used to visualize, debug and improve current adversarial robustness evaluations, providing first concrete step towards automatizing and systematizing current adversarial robustness evaluations.
10
11
12
Introduction
14
Neural networks are now deployed in settings where it is important that they behave reliably and robustly 119 15 [3 33 3. Unfortunately, these systems are vulnerable to adversarial examples [29 4)
16
i... inputs intentionally crafted to mislead machine-learning classifiers test time. These attacks are especially important in settings where classifiers have security-critice consequences, including autonomous driving, automated medical diagnoses, and cybersecurity -elated tasks such spam and
18
20
malware detection, web-page ranking and network protocol verification [27 18 26
28 [5].
21
This vulnerability has caused strong reaction from the community, with many proposed defenses
33
22
22 31 25). Early defenses often argued robustness by showing the defense could prevent prior attacks, but not attacks tailored to that particular defense. As result, most of these defenses have
23
24
turned out to only provide a false sense of security, i.e., to be broken when targeted by an adaptive attack that tailors the attack strategy to the particular defense [I]. More recent work has tried to evaluate using such adaptive attacks. Unfortunately, even this has proven difficult; recent work has shown that 13 published defenses proposed in the last year ineffective despite almost all of them containing an analysis to adaptive attacks [30].
26
28
29
The reason why adversarial example defense evaluations are incomplete comes down to the difficulty of performing an adaptive attack, and diagnosing when they go wrong. Adversarial examples are typically generated through gradient descent: the adversary first constructs loss function so that minimum for that function an adversarial example. While gradient-based attacks are highly effective finding adversarial examples undefended classifiers with smooth loss functions, many defenses substantially hinder the attack optimization by obfuscating gradients or by exhibiting harder-to- optimize loss functions. In particular, most attempted defenses to adversarial examples only succeed at increasing the difficulty of solving the minimization formulation, and not at actually increasing the
30
31
32
34
36
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

37 robustness of the underlying classifier (i.e., increasing the actual distance of the decision boundary Algorithm 1:
Our framework for computing adversarial attacks
Input :a; the initial point; y. the true class of the initial point; n, the number of iterations; Ot the learning rate; f. the target model; A, the considered region.
Output::"* the solution found by the algorithm
1 Zo f initialize(z)
> Initialize starting point
0 <- approximation())
Approximate model parameters
3 ōg < 0
> Initial d
for [1, do
8' 6, aV L(x y:
Compute optimizer step D. Apply constraints (if needed) Choose best perturbation
6+11 c apply--oontrraintsirn 8' A)
7 8" best(ôo: 6,.)
return 5*
38 from the input sample) [T, 1I (0). Moreover, even though guidelines and best practices have
39 been suggested to improve current adversarial robustness evaluations, the lack of automatic testing and debugging tools makes difficult to apply these recommendations a systematic manner. These difficulties have perpetuated a constant cat-and- mouse game where defenders propose new schemes, and attackers find that actually the defense was only increasing the difficulty of solving the underlying minimization problem 3).
40
41
42
44
This paper directly addresses these limitations by (i) developing quantitative indicators of failure, i.e., metrics designed to help debug optimization of gradient-based attacks for generating adversarial examples, and (ii) suggesting systematic evaluation protocol to improve current robustness eval- uations by applying a sequence of specific mitigation strategies. In four case studies of published defenses that have been shown to be ineffective against stronger adaptive attacks, we show (i) that our indicators would have highlighted different failure modes in the original evaluations, and (fi) how these failures could have been easily overcome by following our suggested mitigation strategies.
45
46
47
48
50
51 To summarize, we make the following contributions: (i) we introduce a unified attack framework
52 that captures the predominant styles of existing gradient-based attack methods, and allows us to categorize the five main causes of failure that may arise during their optimization (Sect. 2 (ii) we
53
54 propose five indicators of attack failures (IoAF), i.e., metrics and principles that help understand why attack algorithms fail (Sect. (3); (iii) we empirically evaluate the utility of 56 our metrics on four recently-published defenses, showing how their robustness evaluations could have been improved by monitoring the loAF values and following our evaluation protocol (Sect. 4 and (iv) we provide open-source code and data we used this paper for reproducing resources. Our code is available at /tttps:/githhi
55 and when gradient-basee
58
com/ioaf..... /ioffoof--/oo todo We conclude by discussing related work (Sect. 5) along with the limitations of our work and future research directions (Sect. ()).
60
2 Adversarial Robustness: Gradient-based Attacks and Failures
61
82 We argue here that optimizing adversarial examples amounts to solving a multi-objective optimization: min (L(x + 8, y: 0), ((0)pp) SEA
(I)
63 where x E [0, 1] the input sample, y E (l,...,,c) .c either its label (for untargeted attacks) or
64 the label of the target class (for targeted attacks), and 8 E A is the perturbation optimized to have
65 the perturbed sample x' x + 6 misclassified as desired, within the given input domain. The target model is parameterized by 0. The given problem presents an inherent tradeoff: minimizing L
87 amounts to finding an adversarial example with large misclassification confidence and perturbation size, while minimizing (01pp penalizes larger perturbations (in the given l, norm) at the expense of decreasing misclassification confidence Typically the attacker loss L is defined as the Cross-Entropy (CE) loss, or the logit difference
70
Anonymized for submission.
NNote that the sign of L may be adjusted internally in our formulation to properly account for both untargeted and targeted attacks.

---

-1
-2
0.6
20-
W00 W/
0..4
Lng difeeee
Logit dfbrere
30-
(122
0..3
100 herstione
Iroatton 100
(a) Impl. problems.
(b) Non-converging attack (c) Bad local optimum.
(d) Non-adaptive attack.
The four attack failures that can be encountered during the optimization of an attack. The failed attack path is shown in gray, while the successful attack is displayed in black. The point æo is marked with the red dot, the returned point of the failed attack with a red cross, and the successful adversarial point with the green star. The top row shows the loss landscape, as L(x aw +b2,, yı 0). V1 is the normalized direction (z, - zo), while is a representative direction for the displayed case. In the second row we show the value of L(x + 8%, ys: 0) for the evaluated model.
Figure I:
71 Multiobjective problems can be solved by establishing different tradeoff between the given objectives along the Pareto frontier, by either using soft- or hard-constraint reformulations. For example, Carlini- 73 Wagner (CW) ITI is a soft-constraint attack, which reformulates the aforementioned multiobjective unconstrained optimization: mins ||0l|p + min(L(a + y. 0), -x), where the 75 hyperparameters x and e tune the trade-off between misclassification confidence and perturbation objective while constraining the 77 other. They include maximum moxiimum-oonfidncce attacks like Projected Gradient Descent (PGD) 117, which is formulated min& L(x: 6, y: 0) s.t. || | S c, and minimum-none attacks like Brendel-Bethge (DDN) 244, which can be formulated as mins ||0||p s.t. L(x 5, y; θ) < k. In these cases, and i upper bound the perturbation size and the misclassification confidence, respectively, thereby optimizing a different tradeoff between these two quantities.
74 problem
76 size. Hard-constraint reformulations instead aim to minimize
79 (BB) 6 and Decoupling- Seapliin-Direetoo--Nee
81
82 The aforementioned attacks often need to use an approximation i of the target model, since the latter may be either non-differentiable, or not sufficiently smooth (1. hindering the gradient-based attack optimization process. In this case, once the attacker loss has been optimized on the surrogate model O, the attack is considered successful if it evades the target model 0.
84
85
86 Attack Algorithm. According to the previous discussion, even if different attacks minimize different objectives or require different constraints, all of them can be seen as solutions to a common multiob- jective problem, based on gradient descent. Thus, their main steps can be summarized as detailed in Algorithm First, an initialization point (line T needs to be set, and this can be achieved by directly version of it, even sample from the target class [6].
87
89
using the input point æ. Then, the target model is difficult to deal with, or it is non-differentiable, the attacker must chose
91
82 a surrogate model that approximates the real target 0 (line 2 The attack then iteratively updates the initial point searching for a better and better adversarial example (line 4 computing in each iteration one (or more) gradient descent steps (line 5) using the initial point and the perturbation ö; computed so far. Hence, the new perturbation 0 obtained by enforcing the constraints defined in the problem (lineT), that can be updated accordingly to the chosen strategy (23 24 For maximum confidence approaches, the attack can not exit the A region, and samples are projected accordingly on this ball when reaching the constraints. Similarly, we consider minimum distance attacks successful only if they found adversarial examples inside the A region. At the end the iterations, the attacker has collected all the perturbations along the iterations, formalized as the attack path. The final result
93
94
95
96
97
98
99
101 of the algorithm is the the best perturbation contained in the attack path, w.r.t. the loss they are
minimizing (line])
3

---

FAILURES
Implementation problems
Non-converging attacks
Bad local optimum
Non-adaptive attack
Obluscated Gradients
INDICATORS MITIGATIONS
11: False Fallure (Silent Success)
(2: Step Size too small Too few flerations (Break-point angle)
13: Noisy descent (Increasing loss)
14: Numeric problems in the gradients (Zero gradients)
IS: Bad surrogate
(Non-transferabliit)
M2 Increase iterations, adjust step size
M3 Smoother loss approximation
M4 Use multiple restarts
implementation
M5 Adaptive loss
Indicators of Attack Failures. The top row lists the four general failures encountered in gradient-based attacks. The second row lists the Indicators of Attack failures we propose, and the last row depicts possible mitigations that can be applied.
Figure 2:
103
2.1
Attack failures
We now isolate four failures that can be encountered while optimizing adversarial attacks using Algorithm
105
|
and we bound each of them to specific steps of such procedure.
106
f Implementation Problems. If no adversarial examples are found by the attack, it might be possible that the used implementation include errors or bugs. For example, we isolated bug inside the procedure proposed by Madry et al. [17]. The attack as described returns the adversarial example only by looking at the last point of the attack path (line of Algorithm ( as shown in Fig. [a| but would not return an adversarial example if one was found during search and then passed over.
107
108
109
111
F2: Non-converging attack. When performing gradient descent based attacks, a common problem
112 is that attacks do not converge to any local minimum, as shown in Fig. [b]] This problem can be
113 caused by either the setup of the attack, and in Algorithm [ this is reflected on the values of a and n,
114 i.c the step size of the attack, and the number of iterations. If a is too small, the gradient update step not exploring the space (line of Algorithm Ip. while using too few iterations n might cause an early stopping of the attack (line Algorithm An example of this failure can be found in the evaluation of the defense proposed by Buckman et al. 77 where the authors only used steps of
115
116
117
118
PGD for testing the robustness of their defense, or by the one proposed by Pang et al. [21] where the defense has been evaluated with only 10 steps of PGD. Also, this failure might be triggered either by too-large step size, that lead the optimizer to keep overshooting the local minimum, or the presence of gradient obfuscation techniques 31 that alter the gradients of the model to point to random directions, leading the descent to fail.
119
121
122
123
F3 Bad local optimum. Once the attack reached convergence, the computed point might not be adversarial, since the optimizer has reached a region where it can not update anymore the adversarial perturbation, as shown in Fig. Ic There are few reasons that might lead to such failure. One of them is again caused by the presence of gradient obfuscation, where the optimizer is unable to continue the descent, since it arrived in region where the norms of gradients are (nearly) zero (i.e. flat regions), or again because the gradients are noisy, and the optimization lands on bad local optimum (line S of Algorithm|] []. An example of such failure detected inside the defense proposed by Papernot et al. 22), where the model is trained to have signal in correspondence of samples, and producing regions with no gradient all around them. Another reason might be triggered by the choice of the initialization point itself (line I of Algorithm [. that leads the optimizer into region where no adversarial examples can be found. The latter has been detected by the analysis conducted by Tramèr al. 30) against the defense proposed by Pang al. (1I, where different initialization point lead the attack to find a better solution.
124
125
126
127
129
130
131
132
133
135
136
F Non-adaptive attack. The loss function that the attacker optimizes does not match the actual loss
137 of the target system, and this caused by a bad choice of the surrogate model (line of Algorithm I.
as shown in Fig. ld This issue manifests when either the attack is computed on undefended model, and later tested against the defense, or the target model is not differentiable and the surrogate is not really approximating it. Since we consider both cases, differ from the literature, where the
139

---

141
term non adaapiiee has been used only for attacks that were not specifically designed to target a given defense [3). An examples of this failure is found in the defense proposed by Yu al. [32 where the attack has been computed against the undefended model, and then evaluated against the defense later.
142
143
To maximize the likelihood of creating successful attacks and hence avoiding such failures, current recommendations (30] suggest to (i) select the strongest attacks against the model that is being tested; (ii) state the precise threat model being considered; (iii) select the correct hyperparameters for the
145
148
147
attack being used; and (iv) compute charts to understand how the attacks behave by varying the size
148
of the perturbation. Indeed useful, such are only qualitative recommendations that require ad hoc
149
inspection of each failed attack.
3
Indicators of Attack Failure
150
151
In this section we describe our Indicators of Attack Failures, i.e. tests that help an analyst debug a failing attack. Each of these tests outputs a value bounded between 0 and where values towards I implies the presence of the failure described by the test. Informed by the results of the indicators, we propose potential mitigations that can resolve the presence of the detected failure. An overview of such approach can be appreciated in Fig. 2 where we connect failures with the indicators that quantify them, along with possible mitigations.
152
153
154
155
156
157
J Silent Success. This indicator is designed as a binary flag that triggers when the attack is failing, but a legitimate adversarial exam- ple is found inside the attack path, as described by the implementa- tion problem failure (F1).
158
159
160
Neaaeer Loss
161
1 Break-point angle. This indicator is designed to quantify the
162
non-convergence of the attack (F2) caused by the choice of too small hyperparameters. We normalize the loss along the attack path and the iteration, to fit the loss in the domain [o, 1] x [0, 1]. and, ideally, a well- converged loss should approximate a triangle in that domain, as connect the first and the last point in the loss curve, and we conclude the shape by considering the point of the loss curve that is further to such conjunction. We are interested in the amplitude of the basis 3 angle, since it is the one that characterizes the shape of the triangle: when 33 ss π, the triangle is flat, implying that the loss is still decreasing. For this reason, the indicator computes 1 |cos:3, matching such intended behavior. On the other hand, this indicator is close when the triangle is close to be right, hence 3 Rs z.
163
165
166
shown in Fig. To create that triangle,
Itteraioons
167
Figure 3: 12 indicator.
168
189
170
171
172
173 1: Increasing loss. This indicator is designed to quantify either the non convergeee of the attack (F2), the inability of converging to good local optimum (Fz), both caused by the presence of noisy gradients, where the loss of the attack is increasing while optimizing. To characterize such behavior, we normalize the loss of the attack
175
176
Attecker Loss
177
178
and the iterations as we did in I2, and we extract from it only the portions where it increases, and we compute its area, as shown in
179
180
Fig. When this indicator is close to 1, the values of the loss are fluctuating around its maximum value, difficult to be decreased by the optimizer.
181
182
Iterations
183
4: Zero gradients. This indicator is designed to quantify the
184
bad-local optimum failure (Fz), caused by the absence of gradi- ent information. For this reason, we compute how many times, along the attack path, the gradients of the loss function zero: . Ei 11 Vive.,, =|==- This indicator is close to when most of the norms of the gradient are 0,
Figure 4: 13 indicator.
185
186
187
188
causing the attack step to fail.
1s: Non-transferabilityy This indicator is designed to quantify the non-adaptive failure (F4), by measuring if the optimized attack fails against the real target model, while succeeding against the surrogate one. If the attack transfers successfully, the indicator is set to O, otherwise it is set to 1.
190
5

---

192
3.1
Mitigate the Failures of Security Evaluations
193
Once the robust accuracy of a model has been computed, the attacker should now check the feedback of the indicators and mitigate accordingly the detected failures.
194
195
Mj: Fix the implementation. / is active, the attack is considered failed, but there exists an adversarial point inside the computed path that satisfies the attack objective. Hence, the resulting robust accuracy must be lowered to reflect this patch accordingly. Also, the attacker would want to
196
197
run again their evaluations using another library, or patched version of the same attack.
199
My: Tune the hyperparameters. If /2 activates, it means that the optimization can be improved, and hence both the step size and iteration hyperparameters can be increased. Otherwise, if L activates,
201
the attack should consider a smaller step size, since the loss might be overshooting local minima.
202
My: Use different loss function. If Iy activates, and the decrement of the step size did not work,
203
the attack should change the loss to be optimized [30, preferring one that has a smoother behavior. If
204
4 activates, the attack should consider loss functions that do not saturate (e.g. avoid the softmax) (
205
or also increase the step size of the attack to avoid regions with zero gradients.
206
My: Consider different restarts for the attack. If 13 1 activates, the attack might also consider
207
to repeat the experiments with more initialization points and restarts, as the failure could be the result
208
of added randomness or an unlucky initialization.
209
Mf: Perform adaptive attacks. Lastly, if none of the above applied, the attack might be optimizing against a bad surrogate model. If Is is active, the attack should be repeated by changing the surrogate to better approximate the target, or include the defense inside the attack itself [30]. This step implies repeating the evaluation, as the change of the surrogate might trigger other previously- -fxxd failures.
210
211
212
213
When attacks fail even after the application of recommended mitigations, it would be easy to assume that the evaluated defense is strong against adversarial attacks. However, the only thing known is that baseline attacks, properly tested, are not working against the defense. Hence, the designer of the
215
216
defense should try as hard as possible break the proposed defense with further investigations (12), and by performing sanity checks, e.g., ensuring that the robust accuracy drops to 0% when the perturbation size is unbounded, or by trying different attack strategies, e.g., using gradient-free attacks or attacks designed by reversing the defense mechanism.
217
218
219
Experiments
220
221
We now exhibit the results of our experiments, by showing the correlation between the feedback of indicators, and the false sense of security given by badly-evaluated defenses.
222
Experimental setup. We run our attacks on an Intel Xeon 2 CPU E5-2670 v3, with 48 cores, 126 GB of RAM, and equipped with an Nvidia Quadro M6000 with 24 GB of memory. All the attacks and models have been wrapped and run by using the SecMI. library (20). We select four defenses that have been reported as failing, and show that our indicators would have detected such evaluation errors. For each of them, we set the hyperparameters for the attack as done in the original evaluation, in order to collect similar results.
223
225
227
229 k- -inners-Take-All (KWTA), the defense proposed by Xiao et al. 31 uses only the top-k outputs from each layer, generating many discontinuities in the loss landscape, and hence resulting in the
231
non converngg failure due to noisy gradients (F2). We use the implementation provided by Tramer
232
al. 30), trained on CIFARIO, 10, and we test its robustness by attacking it with &g [17] with step size of a = 0.003, maximum perturbation e = 8/255 and 50 iterations, with 5 restarts for each attack, scoring a robust accuracy of 58% on 100 samples.
233
234
235
Distillation, the defense proposed by Papernot et al. (22), works by training a model to have zero gradients around the training points, leading gradient- based attacks towards bad local optimum (Fz). We re-implemented such defense, by training a distilled classifier on the MNIST dataset to mimic the original evaluation. Then, we apply -PD [17], with step size a 0.01, maximum perturbation
236
237
238
239
= 0.3 for 50 iterations on 100 samples, resulting in a robust accuracy of 94,2%.
240
Ensemble diversity, the defense proposed by Pang et al. (2I] is composed with different neural networks, trained with regularizer that encourages diversity. We adopt the implementation provided
241
by Tramèr et al. 30). Then, following its original evaluation, we apply PGG [17], with step size
242
0.001, maximum perturbation = 0.01 for 10 iterations on 100 samples, resulting in robust accuracy of 38%
244
Turning Weakness into a Strenght (TWS), the defense proposed by al. 32 applies mechanism

---

Model
Attack
12
13
14
15
I
RA
PGD
0.33
0.43
0.77
0.306
58,2%
k-WTA 31
APGD
0.310
0.33
0.128
36,4%
PGD*
0.07
0.48
0.55
0.220
6,4%
PGD
0.98
0.97
0.39
94.2%
Distillation 22
APGD
0.4
0.21
0.122
00.4%
PGD*
0.04
0.008
0%
PGD
0.76
0.152
38%
Ensemble Div. 21
APGD
0.370
0.14
0.102
0%
PGD*
0.08
0.17
0.15
0.080
PGD
0.49
0.07
0.37
0.186
35%
TWS 32
APGD
0.41
0.09
0.10
0%
PGD*
0.37
0.10
0.094
0%
Table 1: Values of the Indicators of Attack Failures, computed for all the attacks against all the evaluated models. We denote the attacks that apply also the mitigations PGD*.
for detecting the presence of adversarial examples on top of undefended model, measuring how much the decision changes locally around a sample. Even if the authors also apply other rejection mechanisms, take into account only the described
247
wish to show that attacks optimized neglecting such term will trigger the non-adaptive attack failure (F4). We apply this defended on
249
WideResNet model trained on CIFARIO, provided by RobustBench [4.. We attack this model with -PG [7]], with step size a = 0.1, maximum perturbation c = 0.3 for 50 iterations on 100 samples, and then query the defended model with all the computed adversarial examples. While the attacks works against the standard model, some of them are rejected by the defense, resulting in robust accuracy of 35%, highlighted by the trigger of the indicator. In this case, we consider an attack unsuccessful if the original sample is not misclassified and the adversarial point is either belonging the same class, it is labeled as rejected.
251
253
254
255
256
257
Each of these attacks have been executed with 5 random restarts. We also attack all these models with the version of AutoPGD (APGD) [3] that uses the difference of logit (DLR) as a loss optimize. This strategy will take care to automatically tune its hyperparameters while optimizing, reducing possible errors that occur while deciding the values of step size, and iterations. Lastly, we compute attacks that take into account all the mitigations we prescribed, and they will be analyzed further on the paper.
258
259
260
262
263
dentiffyng failures. We want now to understand if our in- dicators correlated with faults of the security evaluations of defenses. We collect the results of all the attacks against the selected targets, and we compute our indicators, by listing their values in Table [ along with their mean score. With glance, is possible
264
Corr: 0.86 p-value: 3.51e-04
285
k-WTA
286
Dosilatiio
287
268
grasp that out hypothesis is right:
Rookuet Aceaney
Emsenite Divesity
289
the detection of failure is linked with higher values for the robust accuracy, and also the opposite. Each original evalua- tion is characterized by high values of one more indicator, while the opposite happens for stronger attacks. For instance,
E6
270
6.4
271
272
APGD automatically tunes its hyperparameter while optimiz- ing, hence is able to apply some mitigations directly during the attack. To gain a quantitative evaluation of out hypothe- sis, we compute both the p-value and the correlation between the average score of the indicators and the robust accuracy, depicting this result in Fig. S Both p-value and correlation suggest strong connection between these analyzed quantities, confirming our initial belief.
x
274
0.0
0.0
0.1 Average Metries
0.2
0.3
0.4
276
278
Figure 5: Evaluation of our metrics for different models. Robust accu- racy vs. average value of the indi- cators, for the initial evaluation (de- noted with 'o'), with the evaluation after-mitigation (denoted with 'x'), and with APGD (denoted with *x'
280
281
Mitigating failures. We can now use our indicators to im-
282
prove the quality of the security evaluations, and we apply the following pipeline: (i) we test the defense with a set of points with the original attack strategy proposed by the author
284
of the defense: (ii) we select the failure cases and inspect the
7

---

Model
Initial
M,
M2
M,
M4
M5
Final
k-WTA [1
58.2%
36.4%
36,4%
6.4%
6.4%
6.4%
6.4%
Distillation 22
94.2%
94.2%
94.2%
94.2%
94.2%
0.4%
0.4%
Ensemble Diversity 21
38.0%
38.0%
36.0%
36.0%
29.0%
9.0%
9.0%
TWS
32
35.0%
35.0%
35.0%
35.0%
35.0%
0.0%
0.0%
Table 2: Robust accuracies (%) after patching the security evaluations with the preseribed mitigations.
286
feedback of our indicators per-sample; (iii) for each cause of failure, we apply the specific remediation suggested by the metric; and (iv) we show that the attack now succeeds, thus reducing the robust accuracy of the target model, and also the values of the indicators.
286
289
We report all the results of this process in Table 2 where each row shows the original robust accuracy. and how it is decreased, mitigation after mitigation. Also, all the individual values of each indicator computed on these patched attacks can be found in Table 1 marked as PGD"
290
Mitigating k- WIA failures. For many failing attacks, the // indicator triggers, implying that the attack found an adversarial example inside the path. We then apply mitigation M, and we lower accordingly the robust accuracy of the model to 36,4%. We then analyze the feedback of the I indicator, the one that detects the presence of noisy gradients. We apply mitigation Mi. and we
293
284
285
286
change the loss of the attack as described by Tramèr et al. (30) This loss is computed by averaging the gradient of each single point of the attack path with the information of the surrounding ones. The resulting direction is then able correctly descent toward minimum. We run foe -PGD with the same parameters, but smoothing the gradients by averaging 100 neighboring points from normal distribution N[u ; a 0.031). where z; is a point the attack path. After such mitigation, the robust accuracy drops to 6, 4%, and so follows the indicator (Fig. 6a).
297
298
299
300
302
Mitigating Distillation failures. All the attacks fail because of the absence of gradient information, leading the attack to a bad local optimum (F;), and such is highlighted by the feedback of the I indicator. We apply mitigation M,, and we change the loss optimized during the attack, following the strategy applied by Carlini et al. [9]. that computes the loss of the attack on the logit of the model rather than the final softmax layer. We repeat the PGD attack with such fix, and the robust accuracy drops to 0%, along with the indicator 15 (Fig. 6b).
303
304
305
306
307
Mitigating Ensemble diversity failures. Firstly, the 1, indicator highlighted the presence of Fj, implying that some failing attacks are due to the implementation itself. We apply mitigation M,, and the robust accuracy decreases to 36% Also, lz indicator is active, implying that the loss of of failing attacks could be optimized more. For this reason, we apply mitigation M2, and we increase the step
309
310
311
312
size to 0.05 and the iterations to 50. This patch is enough for lowering the robust accuracy to 9%. (Fig. 6ch.
313
Miligating TWS failures. The detector is rejecting adversarial attacks successfully computed on the undefended model, triggering the Is indicator. Hence we apply mitigation Ms, and we adapt the attack to consider also the rejection class. This version of PGD minimizes the usual loss function of the attacker, but it also minimizes the score of the rejection class when encountered, allowing it to evade the rejection. We such attack, and we obtain a new robust accuracy of 0% (Fig. [d).
315
317
Related Work
319
320
Other systematic analysis on robustness evaluations. There have been number of prior papers evaluating the robustness of particular defense schemes [T0] IT 30). These papers focus on under- standing whether the robustness claims of particular defenses are true, often by performing one-off attacks or by proposing new general attack approaches that can be used to break future defenses. In
321
322
323
324
contrast our goal is not to break any particular defense, but rather to help researchers understand
325
when their evaluation may have gone wrong. In this way our paper is related to Carlini et al. [12 that systematizes various suggestions from the literature for how to ensure that adversarial robust- ness evaluations are performed thoroughly. We imagine that our tests could be included in future recommendations for robustness evaluations.
327
8

---

Init. oval.: 58.2
Init. oval.: 94.2
Init. eval.: 38.0 Final eval.: 9.0
Init. eval.: 35.0
Final eval.: 6.4
Final eval.: 0.4
Final eval.: 0.0
(a) k- WTA
(b) Distillation
(c) Ensemble Div.
(d) Rejection
1: Silent Sucoess
Zz: Break-Point Angle
Iz: Increasing Loss
4: Zero Gradients
Js: Non-transferability
Figure The values our indicators and the success rate (SR) of the attack, before aeemi-transpante colored area) and after (solid colored area) fixing the failures, computed for the analyzed models.
Related to this work, there are a number of attack benchmarks that have been constructed. Instead measuring the robustness of individual schemes as the prior papers do, these benchmarks aim to provide a complete evaluation framework that can be applied to any future defense as well. Ling et. al [16] proposed DEEPSEC, a benchmark that tests several attacks against a wide range of defenses. However, this framework was shown to be flawed by several implementation issues and problems the configuration of the attacks [8] Croce et al. [44 propose RobustBench [[4), that accepts state-of-the-arr models as submissions, and it tests their robust accuracy by applying AutoAttack [3]. However, this benchmark suite only works on CIFAR-trainee models, and not able to determine which are the possible causes of such scored performance.
Benchmarks.
330
332
334
336
338
Hence, these benchmark would benefit from our indicators, since they might provide useful insight that can be autonomously computed. Here we imagine that our framework could be used to help these tools automatically detect when their evaluations are incomplete, so that they could warn the operator that there was potential error that should be investigated.
340
Contributions, Limitations and Future Work
342
343
We propose the Indicators of Attack Failures (IoAF), quantitative tests that help the debugging of faulty-conducted security evaluations, and we propose a pipeline for mitigating their issues, leading to fairer evaluation. We select defenses that have been previously shown to be weak against adversarial attacks, and we evaluate them with the lens of our indicators, showing that we could have detected their misconduct in advance. We empirically prove that these test are correlated with wrongly high robust accuracy, while they drop when attacks are successful.
344
346
348
349
On top of these contributions, we acknowledge some limitations in our methodology. We do not provide aully-autonomous way for deciding how to turn an attack into its adaptive version against particular defense (e.g. gradient obfuscation), but we provide quantitative tools for helping the decision among the possible solutions that the attacker could come up with. Another limitation lurks in the choice of the attack itself, since some unknown-and-adang addatiie attack could behave very differently w.r.t. standard one, triggering some indicator the process. However, these tests can be patched accordingly to take care of these newly-proposed patched attacks, and still being used as debugging tools. Lastly, as already discussed in Sect. 3 if the evaluated defense is not triggering any indicators it does not imply it is secure, but rather it forces the application of other sanity checks [22] We believe some part of this last process can be automatized with additional indicators, however we leave this future work.
350
351
352
354
356
358
360
We hope that future work will include our indicators during the evaluation phase of new methods, in order to identify when attacks are failing for known reasons, and thus contributing to the creation of better defense mechanisms. Also, this work pose a preliminary step towards the creation of interactive dashboards that can be inspected as web application. Finally, it would be insightful to attach our pipeline of indicators and mitigations to already-availahi benchmarks (i.e. RobustBench 14), possibly detecting other failures in security evaluations we did not covered in our experiments.
362
364

---

References
386
367 [1] A. Athalye, N. Carlini, and D. A. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, volume 80 of JMLR Workshop and Conference Proceedings, pages 274-283. Jog..org, 2018.
369
370
[2] M. Barreno, B. Nelson, A. Joseph, and J. Tygar. The security of machine learning. Machine Learning, 81:121-148, 2010.
[3] B. Biggio and F. Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84:317-331, 2018.
373
[4] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šindić, P. Laskov, G. Giacinto, and F, Roli. Evasion attacks against machine learning at test time, In H. Blockeel, K. Kersting, S. Nijssen, and F. Zelezný, editors, Machine Learning and Knowledge Discovery in Databases (ECMI. PKDD), Part III, volume 8190 of LNCS, pages 387-402. Springer Berlin Heidelberg, 2013.
375
376
377
[5] B. Biggio, G. Fumera, and F. Roli. Security evaluation of pattern classifiers under attack. IEEE Transactions on Knowledge and Data Engineering, 26()))98499)) April 2014. ISSN 1041-4347.
379
[6] W. Brendel, J. Rauber, M. Kümmerer, I. Ustyuzhaninov, and M. Bethge. Accurate, reliable and fast robustness evaluation, 2019.
382
383
[7] J. Buckman, A. Roy, C. Raffel, and I. Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations, 2018.
88] N. Carlini. A critique of the deepsec platform for security analysis of deep learning models, 2019.
386
387
[9] N. Carlini and D. Wagner. Defensive distillation is not robust to adversarial examples, 2016.
388
[10] N. Carlini and D. A. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In B. M. Thuraisingham, B. Biggio, D. M. Freeman, B. Miller, and A. Sinha, editors, 10th ACM Workshop on Artificial Intelligence and Security, AlSec '17, pages 3-14, New York, NY, USA, 2017.ACM,
390
[11] N. Carlini and D. A. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pages 39-57. IEEE Computer Society, 2017.
393
394
[12] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Kurakin. On evaluating adversarial robustness, 2019.
385
396
[13] Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse
387
parameter-free attacks. In ICML, 2020.
398
[14] F. Croce, M. Andriushchenko, V. Sehwag, N. Flammarion, M. Chiang, P. Mittal, and
399
M. Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
400
401
[15] S. G. Finlayson, J. D. Bowers. J. Ito, J. L. Zittrain, A. L. Beam, and I. S. Kohane. Adversarial attacks on medical machine learning. Science, 363(6433) 2019.
402
403
[16] X. Ling, Ji, Zou, J. Wang, C. Wu, B. Li, and T. Wang. Deepsee: A uniform platform for security analysis of deep learning model. In 2019 IEEE Symposium on Security and Privacy (SP), pages 673-690, 2019. doi: 10.1 109/SP.2019 00220099.0000 9.00023.
404
405
406
[17] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
408
[18] A. McCallum and K. Nigam. A comparison of event models for naive bayes text classification. In Prec. AAAI Workshop on learning for text categorization, pages 41-48, 1998.
409
410
[19] McDaniel, N. Papernot, and 7. B. Celik. Machine learning in adversarial settings. IEEE Security & Privacy, 14(3):68-77 May 2016.
10

---

412
[20]
M. Melis, A. Demontis, M. Pintor, A. Sotgiu, and B. Biggio. secml: A python library for secure and explainable machine learning, arXiv preprint arXiv:1912.10011 2019.
413
414
[21]
Pang, K. Xu, C. Du, N. Chen, and Zhu. Improving adversarial robustness via promoting ensemble diversity. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseurch, pages 4970-4979. PMLR, 09-15 Jun 2019, URL htlp:ttp//// htttp://proccedingg. mlr press/v97/pangl html
415
416
418
419 [22] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation defense adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pages 582-597, May 2016. doi: 10.11099SP.01..11.
421
422
[23] M. Pintor, F. Roli, W. Brendel, and B. Biggio. Fast minimum-norm adversarial attacks through adaptive norm constraints, 2021.
[24] J. Rony, L. G. Hafemann, I.. S. Oliveira, I. B. Ayed, R. Sabourin, and E. Granger. Decoupling direction and norm for efficient gradient-based adversarial attacks and defenses, 2019.
425
426
[25] K. Roth, Y. Kilcher, and Hofmann. The odds are odd: statistical test for detecting adversarial examples. In International Conference Machine Leurning, pages 1498-55077 PMLR, 2019.
428
429
[26] B. I. Rubinstein, B. Nelson, L. Huang, A. D. Joseph, S.-h. Lau, Rao, N. Taft, and J. D. Tygar. Antidote: understanding and defending against poisoning of anomaly detectors. In Proceedings of the 9th ACM SIGCOMM Internet Measurement Conference, IMC '09, pages 1-14, New York, NY, USA, 2009. ACM.
431
[27]
M. Sahami, S. Dumais, D. Heckerman, and E. Horvitz. A bayesian approach to filtering junk e-mail. AAAI Technical Report WS-98-05, Madison, Wisconsin, 1998.
434
435
[28] C. Smutz and A. Stavrou. Malicious pdf detection using metadata and structural features. In Proceedings of the 28th Annual Computer Security Applications Conference, ACSAC 12, pages 239-248, New York, NY, USA, 2012. ACM.
437
438
[29] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intrigu- ing properties of neural networks. In International Conference on Learning Representations, 2014. URL /ttpp://axxiv /twxxv..ivv////////////////.a/ org/ abs a/1// 1312, 6199
440
441
[30] Tramer, N. Carlini, W. Brendel, and A. Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 33, 2020.
[31] C. Xiao, P. Zhong, and C. Zheng. Resisting adversarial attacks by A-winners-take-all 2020.
[32] T. Yu, S. Hu, C. Guo, W. Chao, and K. Weinberger. A new defense against adversarial images: Turning a weakness into a strength. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Oct. 2019.
445
[33] X. Yuan, P. He, Q. Zhu, and X. Li. Adversarial examples: Attacks and defenses for deep learning. IEEE Transactions on Neural Networks and Learning Systems, 30(9):2805-25) 2824, 2019. doi: 10.1119/TNNLSS2011 2018 2011..81..
448
Checklist
450
1. For all authors...
452
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
453
454
(b) Did you describe the limitations of your work? [Yes] We discuss the limitations in Sect. 6
455
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In Sect. 6 we specified that the aim of our work is not to break defenses an harmful way. Our
457
purpose is only to help researchers to improve their security evaluation.
11

---

459
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
480
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
485
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] The code will be submitted as supplementary material, and the instructions for reproducing the experiments are described in Sect. 4
486
487
468
489
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We deseribe the experimental protocol in Sect. +
471
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [N/A]
472
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The resourced used for the experiments are listed in Sect. 4
474
475
478
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
477
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited all the existing assets used for the experiments.
479
(b) Did you mention the license of the assets? [Yes] We cited the authors of the assets, and we provide the list of external assets along with the code.
480
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We provide the code for computing the metries as supplementary material.
482
483
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] All the assets we used are publicly available.
484
485
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
486
487
5. If you used crowdsourcing or conducted research with human subjects...
488
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
480
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
481
482
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
493
12


Reviews:
[{"review_id": "vX20yCKbbus", "reviewer": "Reviewer_9wg3", "summary": "The paper, similar to [1], proposes a systematic way to debug problems that may arise when implementing adversarial attacks to evaluate a proposed defense. They categorize the implementation problems into 4 categories, propose 5 indicator metrics that can identify what problem exists in an implementation, and highlight 5 mitigation strategies that can be used when a particular indicator is triggered. In the context of 4 existing defenses, they show that original Projected Gradient Descent attack implementation overestimates the security of these defenses. By leveraging the proposed indicator values and using their suggested mitigations to properly generate PGD adversarial examples, they are able to show that the attack is much more lethal and can invalidate the existing defenses. They compare the efficacy of their manual approach for debugging with AutoPGD, which tunes hyperparameters automatically to improve the PGD attacks chances of success.\n\n[1] Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., ... & Kurakin, A. (2019). On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- The paper is easy to follow.\n  1. Provides an abstract view of adversarial attacks and shows where implementation downfalls arise,\n  2. Proposes a set of indicators to identify these downfalls,\n  3. Suggests which existing mitigation strategies to leverage when a indicator triggers,\n  4. Shows the efficacy of this manual process to debug attack implemenation by comparing it to automatic approaches.\n\n- The paper treats a subset of the issues highlighted in [1] in a quantitative fashion by proposing indicator functions.\n\n\n### Weaknesses / Correctness / Clarification\n\n- The main novelty of the paper seems to be proposing the indicator functions. Note that the universal skeleton of attack algorithms and the mitigation strategies are already known. Perhaps, tying them together in the context of the indicator functions (i.e. parts of the algorithm that trigger it and mitigations that need to be used) is another novelty.\n\n- On the indicator functions, I have a few concerns, clarifications and perspectives. As an overall comment, I suggest you spend more time describing the challenges you faced to come up with these metrics, describe them with more rigor and highlight how it can take care of edge scenarios (Many of the figures/tables have redundant information; hence, space should not be a problem). More specifically,\n  - I_1 (silent success) and I_5 (non-transferability) are zeroth-order strategies that are often used by researchers who try to debug the efficacy of an attack algorithm. While these simple measures do show that not all researchers did due diligence when evaluating defenses, I am not convinced that these indicators address any challenge-- it is good software development guidance.\n  - For I_2 (break-point angle), the authors state \"we conclude the shape by considering the point of the loss curve that is further to such conjunction\" (lines 168-169). I unfortunately did not understand this. I wanted to evaluate my understanding on some cooked up complicated loss plots but couldn't as I wasn't sure how to concretely identify this third vertex from the statement above.\n  - For I_3 (increasing loss), is there some sort of formal way to say that this measure of area will always be (an admissible) heuristic for evaluating if the attack can converge to a good local optima? While for figure 4 it does make sense, I am not sure if it is always going to work. Even a proof for a weaker statement like _Given two attacker loss plots (for the same model, data blah..), if one converges to a better local optima than the other, the value of I_3 will always be smaller for it_ should work.\n  - For I_4 (zero gradients), again some sort of formal treatment + highlight what was the major challenge one needed to solve to come up with it would be helpful.\n\n- From table 1, I see that APGD has higher values for several indicator functions, such as Distillation I_2 (0.4), Ens. div. I_2, TWS I_2 then the mitigated PGD*. Yet, in all 3 cases, Auto PDG is almost similar or better than PGD*. This makes me question the validity of I_2 as a good indicator. Similarly for I_3, APGD has almost the same value as Ensemble Div for APGD (0.14) and PGD* (0.15) and yet APGD is much more effective at mitigating implementation issues. As stated above, without some formal guarantees, it is hard to accept them as indicators on face value.\n\n- Evaluation against AutoPGD shows that this manual effort is better only in the context of debugging attack implementation for k-WTA. For others, its almost it's either a bit worse (ensemble div) or similar (distillation, TWS). At this point, I am not convinced why should one rely on manual debugging techniques (which may have pitfalls of their own).\n\n- As all gradient based attacks (eg. CW, PGD, BB, DDN) can be represented as part of their framework, wondering why evaluation only looks at PGD? They should create an auto-version of the other attacks and adapt them as baselines to compare against. Simply showing efficacy of indicators based on only one attack type (i.e. PGD) isn't enough.\n\n- The indicators provide a user confidence that many obvious issues do not exist but is not sufficient to say that having mitigated all the issues with the attack implementation, if the defense is really resilient (217-219).\n\n- Table 1 shows a subset of indicators are used for each defense-attack combo, but Table 2 (sort of) implies all mitigations are applied in a step-by-step fashion. Not sure if a particular sub-set of indicators are triggered, why are all mitigations applied? If not, please ensure the tables don't mislead the reader.\n\n- The (pretty) figures (Fig 5 and Fig 6) seem redundant. All of this info is already there in the tables (or am I missing some critical insight here?).\n\n#### Minor\n\n- (245) Streng`ht` -> Streng`th`\n- (Table 1) Replace `,` with `.` in accuracy values 58`,`2% -> 58`.`2%\n- (249) defen`ded` -> defen`ces`\n- (301) 6`,`4% -> 6`.`4%\n\n\n[1] Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., ... & Kurakin, A. (2019). On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "dxzhjp0DgS", "reviewer": "Reviewer_wRYi", "summary": "This paper reasons about complications that may cause authors to not recognized that their newly proposed defense is ineffective. The authors identify failure motivations, propose 5 indicators (measurable) that are indicative of an attack failure, and also propose appropriate mitigations. The paper very methodically and experimentally presents this findings, and offers a more concrete support towards adaptive attacks evaluations whenever new defenses are proposed. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- Very important line of research, focused on supporting and understanding how to systematically carry out adaptive attacks.\n- Contributions, significance and novelty is clear.\n- Systematic evaluation of indicators of attack failures in existing defense robustness evaluations\n- Code scheduled for release. \n\n# Weaknesses\n- Mitigation $M_5$ seems a bit vague in terms of actionable points. \n- The code  link seems broken. Not sure if it was just meant as a placeholder for future publication of code?\n- \"Robust accuracy\" has not been formally defined\n- Unclear interpretation of a 'good' or 'bad' value for the indicators fo attack failures (e.g., Figure 1)\n\n# Detailed Comments\n\nThe indicators presented are derived from very nice intuitions and quantify effects.  I think this is a very systematic and well-thought work that can improve the state of the art in terms of best practices when evaluating defenses. I do have some comments that could improve the quality of the work. \n\n## Mitigations\n\n$M_1$ through $M_4$ are quite clear. $M_5$ starts to be a bit fuzzy, but the last paragraph of Section 3 should require more discussion. If I understand correctly, you are suggesting that the whole framework you propose for indicators of attack failure is just for the following:\n\n> the only thing known is that baseline attacks, properly tested, are not working against the defense\n\nWhat can be the utility of the indicators of attack failures if you try to propose a more advanced adaptive attack upon further investigation?\n\n## Experimental evaluation\n\nThe authors conduct a thorough experimental evaluation considering also many existing defenses. However, there are some things that should be better specified:\n- **Definitions**. There is no formal definition of \"Robust accuracy\". \n- **Parameters**. You perform 5 random restarts of the attacks. Is this a number you recommend also future people to use, or is it a good number as long as there are no indicators of attack failures that activate?\n- **Interpretation of the indicators**. In Figure 5, you show the average score of the indicators on the X-axis. What I see is for example that the average scores for k-WTA are higher both before and after mitigations than most of the other scenarios (e.g., TWS and Ensemble Diversity). Moreover, it seems you have an average still >0.2 for k-WTA. How should readers interpret such results, and know whether the average score of the indicators is sufficiently low?\n\n## General comment\n\nI may be playing the devil's advocate here, but it would be interesting to have the following discussion:\n- Do you feel that an attacker may take advantage of the fact that you proposed the indicators of attacks failure?\nIn a sense, can an attacker take advantage of the fact that a defender is using indicators of attack failure to evaluate the robustness of their defense? Maybe an attacker that proposes an apparently robust algorithm but of which he knows some other weaknesses?\n\n## Minor comments\n\n- Figure (1) may also use a different line-style because black and gray look a bit similar to my eyes. Maybe also a different line width? Or a greater color difference?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Mostly comments regarding the interpretation of the average scores, and how the indicators may be useful for further adaptive attacks.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "QM59z3Nx1XE", "reviewer": "Reviewer_Ypvj", "summary": "This paper proposes a systematic method for understanding the failures of adversarial attacks as means towards thorough evaluation of robustness defenses in the research community. The devised framework relies of three connected aspects: (i) analysis of the main causes why gradient-based adversarial attacks fail, (ii) metrics for identifying their reason of failure and (iii) practical mitigations to ensure attack efficiency when evaluating defenses. All these aspects are illustrated and confirmed on four existing defenses that were initially subjected to limited evaluation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity\n- Very clear and well-organized paper.\n- The code for reproducing the experiments was provided.\n\nOriginality:\n- The paper formalizes what is common knowledge in the community. However, I do appreciate the effort that goes into this formalism and the necessity for the community to have such a methodology to rely upon.\n\nQuality and soundness:\n- The motivation of the paper is clear, and I agree that, despite existing guidelines for robustness evaluation, robustness benchmarks in published research are often incomplete or misleading.\n- This work is in line with previous guidelines on robustness evaluation.\n- Previous work in the fields of robustness evaluation, broken defenses and machine learning security evaluation platforms are cited appropriately.\n- The proposed analysis of four existing defenses is thorough and consistent with the proposed evaluation methodology.\n\nSoundness and significance:\n- While I appreciate the proposed framework and its application to the four chosen defenses, a more extensive experimental study would probably be more convincing, as the devised framework is mostly based on observations (as opposed to a formal proof of any kind). There is an abundance of published defenses with incomplete evaluation. Showing the applicability of the contribution to a wider range of defenses would give it better odds of reaching its target: becoming part of the de facto robustness evaluation standard. This suggestion is in line with previous publications proving the lack of robustness in prior methods (e.g., [1, 10] in the paper), where around ten defenses are examined in each.\n\nMinor:\n- Fig. 1b: colours seems to be mixed-up between the successful (shown in grey, but should be black) and failed attacks.\n- Tab. 1: the explanation for the column name RA should be introduced in the table caption or the paragraph commenting the table.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "12WSug9TlSm", "reviewer": "Reviewer_ZPse", "summary": "This paper design four quantitative indicators of attack failures (IoAFs) that might help to systemically automatically check the correctness of robustness evaluations. Besides, the authors also provide several mitigation methods for these IoAFs. According to these indicators (IoAFs) as well as corresponded mitigation methods, the authors show the effectiveness of indicators (IoAFs) applied on several existing defense approaches and also proposed mitigation methods.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1. This work propose quantitative indicators to point out reasons of attack failures.\n2. The experiments show the effectiveness of proposed indicators of attack failures (IoAFs) and mitigation strategies.\n\nCons: \n1. Based on [1], we know that IoAFs can work for the correctness of robustness evaluations. However, it only designs metrics for known factors in this paper. It's not surprising that IoAFs are effective. It would be better if the authors have some non-heuristic or theory-driven reasons to explain why this works.\n2. The proposed mitigation strategies have already been used in several papers. For instances, $M_2$, $M_4$ and $M_5$ were used in [2, 3]. Therefore, it exists several works showing that the proposed mitigation strategies are helpful.\n3. The defense methods included in the experiments were defeated in existing works [4,5].\n\nOverall, quantifying the indicators of attack failure is a novel idea. But the concepts of these indicators have already been proposed before and similar scenario also happened the proposal of mitigation strategies. Besides, the defense methods in the experiments were all defeated in recent works. Namely, these posited mitigation strategies might already exist or at least are pretty common.\n\n[1] Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.\n\n[2] Diversity Can Be Transferred: Output Diversification for White- and Black-box Attacks.\n\n[3] Enhancing Gradient-based Attacks with Symbolic Intervals.\n\n[4] On Adaptive Attacks to Adversarial Example Defenses.\n\n[5] Towards Evaluating the Robustness of Neural Networks.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors listed their limitation and meanwhile proposed the alternative method to address it.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers acknowledged the usefulness of the proposed indicators for debugging attack failures, but at the same time, there are major concerns being raised regarding overlapping contributions to known literature and insufficient experiments for demonstrating the value of the indicators. The authors' rebuttal did not address the main concerns of the majority of the reviewers.

Here is a summary of the main concerns. I hope the authors find the review comments and summary useful for preparing the future version.

1. The introduced indicators are useful in debugging attack algorithms, but many of them are common practices or overlap with known results in existing literature, limiting the contributions of this work
2. Some indicators overlap with existing literature on adaptive attacks, such as the NeurIPS 2020 paper "On Adaptive Attacks to Adversarial Example Defenses"
3. This paper mainly focused on 4 defenses (that were previously shown to be broken) while earlier works of similar fashion such as "Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples" and "On Adaptive Attacks to Adversarial Example Defenses" show defense failures on about 10 defenses.
Decision: Reject