Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples
Authors: Maura Pintor
Luca Demetrio
Angelo Sotgiu
Ambra Demontis
Nicholas Carlini
Battista Biggio
Fabio Roli
Venue: NeurIPS 2022 Accept (2022)
Keywords: Debugging
machine learning
adversarial machine learning

Abstract: Evaluating robustness of machine-learning models to adversarial examples is a challenging problem. Many defenses have been shown to provide a false sense of robustness by causing gradient-based attacks to fail, and they have been broken under more rigorous evaluations.
Although guidelines and best practices have been suggested to improve current adversarial robustness evaluations, the lack of automatic testing and debugging tools makes it difficult to apply these recommendations in a systematic manner.
In this work, we overcome these limitations by: (i) categorizing   attack failures based on how they affect the optimization of gradient-based attacks, while also  unveiling two novel failures affecting many popular attack implementations and past evaluations;
 (ii) proposing six novel \emph{indicators of failure}, to automatically detect the presence of such failures in the attack optimization process; and (iii) suggesting a systematic protocol to apply the corresponding fixes. 
Our extensive experimental analysis, involving more than 15 models in 3 distinct application domains, shows that our indicators of failure can be used to debug and improve current adversarial robustness evaluations, thereby providing a first concrete step towards automatizing and systematizing them. Our open-source code is available at: https://github.com/pralab/IndicatorsOfAttackFailure.

Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples
Maura Pintor
Luca Demetrio University of Genoa, Italy
University of Cagliari, Italy Pluribus One, Italy
Pluribus One, Italy
maura. pintoruunica.it
luca. dewetriiounneee .tt
Angelo Sotgiu
Ambra Demontis University of Cagliari, Italy ambra. demontis@unica.
Nicholas Carlini Google ni micholas@carlinie com
University of Cagliari, Italy angelo. sot.iuuunica..i.... it
Battista Biggio
Fabio Roli
University of Cagliari, CINI, Italy Pluribus One, Italy
University of Genoa, CINI, Italy
Pluribus One, Italy fabio.roli@umg
battista. biggiouunica..aa..oo.. it
Abstract
Evaluating robustness of machine-learing models adversarial examples is a challenging problem. Many defenses have been shown to provide a false sense of robustness by causing gradient-based attacks to fail, and they have been broken under more rigorous evaluations. Although guidelines and best practices have been suggested to improve current adversarial robustness evaluations, the lack automatic testing and debugging tools makes it difficult to apply these recommen- dations in a systematic manner. In this work, we overcome these limitations by: (i) categorizing attack failures based how they affect the optimization of gradient- based attacks, while also unveiling two novel failures affecting many popular attack implementations and past evaluations; (ii) proposing six novel indicators of failure, to automatically detect the presence such failures in the attack optimization process; and (iii) suggesting a systematic protocol apply the corresponding fixes. Our extensive experimental analysis, involving more than 15 models in distinct application domains, shows that our indicators of failure can be used debug and improve current adversarial robustness evaluations, thereby providing a first con- crete step towards automatizing and systematizing them. Our open-source code available at: http://hhh...ccooooooca/aaaatttt
1 Introduction
Despite their unprecedented success in many different applications, maahhne--larringg models have been shown to be vulnerable to adversarial examples [4]] i.c., inputs intentionally crafted to mislead such models at test time. While hundreds of defenses have been proposed to overcome this issue 46
29 43 36), many of them turned out to be ineffective, as their robustness to adversarial inputs was significantly overestimated. existing attacks with inappropriate hyperparameters (e.g., using insufficient number of iterations), while others were implicitly obfuscating gradients, thereby causing the optimization of gradient-based attacks fail. Such defenses have been shown yield much lower robustness under more rigorous robustness evaluations and using attacks carefully adapted to evade them 10 2
particular, some defenses were evaluated by running
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

To prevent such evaluation mistakes, and help develop better defenses, evaluation guidelines and best practices have been described in recent work [T] However, they have been mostly neglected their application is non-trivial; in fact, 13 defenses published after the release of these guidelines were found again to wrongly evaluated, reporting overestimated adversarial robustness values 41 has also been shown that, even when following such guidelines, robustness can still be overestimated [33] Finally, while recently-pri pprepssdp frameworks combining diverse attacks seem to provide more reliable adversarial robustness evaluations [13 44] it is still unclear whether and to what extent the considered attacks can also be affected by subtle failures.
To address these limitations, in this work we propose the first testing approach aimed to debug and automatically detect misleading adversarial robustness evaluations. The underlying idea, similar to traditional software testing, is to lower the entry barrier for researchers and practitioners towards performing more reliable robustness evaluations. To this end, we provide the following contributions: (i) we categorize four known attack failures by connecting them to the optimization process of gradient- based attacks, which enabled us also to identify two additional, never- see--befforeeeen failures affecting many popular attack implementations and past evaluations (Sect. 2.1): (ii) we propose six indicators of attack failures (IoAF), i.c., quantitative metrics derived by analyzing the optimization process of gradient-basee based attacks, which can automatically detect the corresponding failures (Sect. 2.2): and (iii) we suggest a systematic, semi-automated protocol to apply the corresponding fixes (Sect. 2.3) We empirically validate approach on three distinct application domains (images, audio, and malware), showing how monitoring the IoAF values and following our evaluation protocol (Sect. (3). We also open source our code and data to enable reproducibility of our findings at https: /gi siiti thub. cwwmcoomo/eee pralab/ Indi cadicators:he Atttakkkaa lure We conclude by discussing related work (Sect. limitations, and future research directions (Sect. 5).
wrongly-evaluater defenses could have been evaluated correctly by
2 Debugging Adversarial Robustness Evaluations
We introduce here our automated testing approach for adversarial robustness evaluations, based on the design of novel metrics, referred to as Indicators of Attack Failure (IoAF), each aimed to detect specific failure within the optimization process of gradient-based attacks.
2.1 Optimizing paadient-based Attacks
Before introducing the IoAF, we discuss how gradient-based attacks are optimized, and provide a generalized attack algorithm that will help us identify the main attack failures. In particular, we consider attacks that solve the following optimization problem:
min s.t.
L(x + 6, y4: 0):
|00| s, and x + c+ E [0, 11
(2)
where x E [0, 1]d is the d-dimensional input sample, y/ E Y f1, fl,..., c} is the target class label (chosen to be different from the true label y the input sample), E Rd is the perturbation applied to the input sample during the optimization, and O are the parameters of the model. The loss function defined such that minimizing it amounts to having the perturbed sample x + misclassified (CE) loss, or the so-called logit loss [10), i.c., L(x + 6, yt. 0) max fa(x + 8, 0) fv. (x + 8, 0), being fi(, 0) the model's prediction (logit) for class Let us finally discuss the constraints in Eq. (2. While the fp-norm constraint || Se e bounds the maximum perturbation size, the box constraint x + d E [0, 1]4 ensures that the perturbed sample stays withing the given (normalization) bounds.
as yt- Typical examples include the Cross-Entrop
Transfer Attacks. When the target model is either non-differentiable or not sufficiently smooth 2] a surrogate model ô can be used to provide a smoother approximation of the loss L, and facilitate the attack optimization. The attack is thus optimized on the surrogate model, and then evaluated against the target model. If the attack is successful, then is said to correctly transfer to the target RD
Note that, while Eq. 1 describes targeted attacks, untargeted attacks can be easily accounted for by substituting ys with y and changing the sign of L.

---

Algorithm 1: Generalized gradient-based attack for optimizing adversarial examples.
Input :æ; the initial sample; yt, the target class label; the number of iterations; o, the step size; θ, the target model; I. the loss function; IT, the projection operator enforcing the constraints in Eq. (2).
Output:2", the solution found by the algorithm
æ initialize(z)
Initialize starting point Use surrogate model (if required)
2 0 + approxination(f)
> Initialize d
4 for i E [1, do
d; < 8,-1 V
Lax
81 Mt; 0)
Compute gradient (p)ate
d, 6 a( 6;)
D Project 6 onto the feasible domain (Eq. ?
return x* best (ast(dg:... kn)
Return best solution
Generalized Attack Algorithm. We provide here generalized algorithm, given Algorithm which summarizes the main steps followed by gradient-basee -based attacks to solve Problem ( 2 The algorithm starts by defining an initialization point (line [], which can be the input sample æ, mandomly-perturbee version of it, or even a sample from the target class [5]. If the target model is
either non -on-differentiable or not sufficiently smooth, surrogate model O can be used to approximate it, and perform a transfer attack (line 2}. The attack then iteratively updates the point to find an adversarial example (line computing one (or more) gradient updates in each iteration (line while the perturbation 6,+1 projected onto the feasible domain (i.c., the intersection of the constraints in Eq. 2 via projection operator II (line 6). The algorithm finally returns the best perturbation across the whole attack path, i.e., the perturbed sample that evades the (target) model with the lowest loss (Zi This generalized attack algorithm provides the basis for identifying known and novel attack failures, by connecting each failure to specific step the algorithm, as discussed the next section.
Other Perturbation Models. We conclude this section by remarking that, even we consider additive f--norm perturbations in our formulation, the proposed approach can be easily extended to more general perturbation models (c.g.. represented as x' = f(xx, 8), being x' the perturbed feature representation of valid input sample, and h manipulation function parameterized by d, as in (17)). In fact, our approach can be used to debug and evaluate any attack as long as it is optimized via gradient descent, regardless of the given perturbation model and constraints, as also demonstrated in our experiments (A.3,A.4, A.5).
2.2 Indicators of Attack Failure
We introduce here our approach, compactly represented in Fig, ! by describing the identified attack failures, along with the corresponding indicators and mitigation strategies. We describe two main categories of failures, respectively referred as loss- las-saadde and attack-oprimizattonn failures.
Loss-landscape Failures, Mitigations, and Indicators. The first category of failures depends on the choice of the loss function L and of the target model 0, regardless of the specific attack implementation. In fact, it has been shown that the objective function in Eq. (1) may exhibit obfuscated gradients [10 2). which prevent gradient-based attacks find adversarial examples even when they exist within the feasible domain. We report two known failures linked to this issue, referred as F, and F2 below.
F₁: Shattered Gradients. This failure is reported in B 8]. It compromises the computation of the input gradient V V. and the whole execution of the attack, when at least one of the model's components (c.g.. g..& specific layer) is oon-differentiable causes numerical instabilities (F, in Fig. 3. My: Use BPDA. This failure can be overcome using the Backward Pass Differentiable Approximation (BPDA) A1 i.e., replacing the derivative of the problematic components with the identity matrix. 1, Unavailable Gradients. Despite the failure and mitigation being known, no automated, systematic approach has ever been proposed to detect F,. This newly-proposed indicator is able to automatically detect the presence of non-differentiable components and numerical instabilities when computing gradients; in particular, for each given input æ, if the input gradient V T., yt, A) has zero norm, or if its computation returns error, we set L (œ) = 1 (and 0 otherwisc).

---

Loss Landscape (Cotuczated Grastents)
Atlack Optimization
Fallures
Unreachable *
Shattered
Stochastic Gradients
Implementation Errors
Non-converging
Non-adaptive Attack
Gradients
Attack
Misclassiffcation
Indicators
Unavailable
Unstable Predictions
Silent Success
Incomplete Optimization
Transfer Fallure
Unconstrained Failure
Gradients
Mitigations
M
M2
Mr Change Loss (Bad Local Minimum)
Tune Step and Iterations
Change Lass (Adaptive)
Use BPDA
Use EoT
Implementation
Loss/Moods--ppeliicc enaure gradients are smooth
Alfack-specific ensure ettack optimization runs correctly
Figure I: Indicators of Attack Failure (IoAF), We show the connections among the distinct attack failures (top), the proposed loAF (center), and the corresponding mitigations (bottom). The novel failures, indicators, and mitigations found in this work are highlighted in bold and marked with "*".
F2: Stochastic Gradients. This failure is reported in 2 41 The input gradients can be computed, but their value is uninformative, following the gradient direction does correctly minimize the loss function L. Such issue arises when models introduce randomness in the inference process (22) when their training induces noisy gradients 4 in Fig.
My: Use E.T. Previous work mitigates this failure using Expectation over Transformation (EoT) [2]
41 i... by averaging the loss over the same transformations performed by the randomized model, or over random perturbations of the input (sampled from a given distribution).
1z: Unstable Predictions. Despite the failure and mitigation being known, no automated, systematic approach has been considered to automatically detect Fz. This novel indicator measures the relative variability Vxx) the loss function L around the input samples. Given an input sample æ, we draw perturbed inputs uniformly from a small Eg ball centered on with radius and compute V(x) = min( Ei= (Lo /z)//ool 1) E [0, 1], where Lo is the loss obtained at æz, Li, Ltg are the loss values computed on the perturbed samples, and the min operator is used to upper bound V(x) S We then set T²(x)) = (x) )) 2 т (and 0 otherwise). In our experiments, we set the parameters of this indicator as 100, r 10 3 and 10%. explained in Sect.
Attack-optimization Failures, Mitigations, and Indicators. The second category of failures is con- nected to problems encountered when running a gradient-based attack to solve Problem -- with Algorithm [ This category encompasses failures F3-Fs Fig.
F3: Implementation Errors. We are the first to identify and characterize this failure, which affects many widely- attack implementations and past evaluations. In particular, we find that many attacks return the last sample of the attack path (line 7 Algorithm 2. even if it is not adversarial, discarding valid adversarial examples found earlier in attack path (F3s in Fig. 3p. This issue affects: (i) the PGD attack implementations (25 currently used in the four most-used Python libraries for crafting adversarial examples, i.e., Foolbox, CleverHans, ART, and Torchattacks (for further details, please refer to A.6); and also (ii) AutoAttack [13), as it returns the initial sample if adversarial example is found, leading to flawed evaluations when used in transfer settings (55.
My: Fix Attack Implementation To fix the issue, we propose modify the optimization algorithm to return the best result. This mitigation can also be automated using the same mechanism used to automatically evaluate the loAF, i.e., by wrapping the attack algorithm with function that keeps track of the best sample found during the attack optimization and eventually returns it.
I;: Silent Success. We design (ar) as binary indicator that set to 1 when the final point in the path is not adversarial, but an adversarial example is found along the attack path (and 0 otherwise).
F4: Non-converging Attack. This failure is reported in [41, noting that the flawed evaluations performed by Buckman et al. [6] and Pang et al. (28), respectively, used only 7 and 10 steps of PGD for testing their defenses. More generally, attacks may not reach convergence (F4 in Fig. 3 due to inappropriate choices of their hyperparameters, including not only an insufficient number of iterations n (line 4 of Algorithm D]. but also an inadequate step size (line of Algorithm

---

Figure 3: The six attack failures that can be encountered during the optimization of attack. The failed attack path is shown in red, while the successful attack is displayed in black. The point Do is marked with the red dot, the returned point of the failed attack with red cross, and the successful adversarial point with the green star. The top row shows the loss landscape, as L(x awv+av bv2, ykü 0). Lap 00400
1000
MA
Leg
Los 0:21
1n0
30
Berstions
Ierooionn
Isorations
Ieerat
Ineoiione
v1 is the normalized direction (z:, σo), while v2 is a representative direction for the displayed case. In the second row we show the value of ((x + 8;, ys: 0) for the evaluated model.
My: Tune Step Size and Iterations. The failure can be patched by increasing either the step size a or the number of iterations n. 4: Incomplete Optimization. To automatically evaluate if the attack has not converged, we propose a novel indicator that builds a monotonically decreasing curve L for the loss, by keeping track of the best minimum found at each attack iteration (a.k.a. cumulative minimum curve, shown in black Fig. 2 Then, we compute the relative loss decrement (xx) in the last A iterations (out of n) as: D(x) (m) i 1/o-=)//mmaa (00 min, (9) [0. 1]. We set [(x)) = if ((x) 2 p (and 0 otherwise). In our experiments, we the parameters
Loss
Iterations
10 and μ 1%, detailed in Sect. 3
Figure 2: 14 indicator.
Fs: Non-adaptive attack. This failure only qualitatively discussed in 3 11 41, showing that many previously-e serriuslyypublliheed defenses can be broken by developing adaptive attacks that specifically target the given defense mechanism. However, to date, it is still unclear how to evaluate an attack is really adaptive below. We first note that this issue arises when existing attacks are executed against defenses that may either have non-differentiable components [6] cases, the attack implicitly executed a transfer setting against surrogate model which retains the main structure of the target, but ignores the problematic components/detectoor:; e.g., Das et al. [16] simply removed the JPEG compression component from the model. However, it should be clear that
not (as it is mostly left to subjective judgment). We try to shed light this issue
additional detection mechanisms (26] In both
optimizing the attack against such a bad surrogate Ô (line 2 of Algorithm is not guaranteed to also bypass the target model 0 (Fs in Fig. Bp.
M5: Change Loss (Adaptive). Previous work 3 approximations of the target, i.e., loss functions and surrogate models that consider all the components of the defense in the attack optimization. This is unfortunately non-trivial step to automate.
applied custom fixes by using better
Is: Transfer Failure. Despite the failure and some qualitative ideas implement mitigations being known, it is unclear how measure and evaluate if an attack is really adaptive. To this end, we propose an indicator that detects F5 by evaluating the misalignment between the loss L optimized by the attack on the surrogate Ô and the same function evaluated on the target We thus set 4s (x) = 1 if the attack sample bypasses the surrogate but not the target model (and 0 otherwise).
Fs: Unreachable Misclassification. We are the first identify and characterize this failure. assumes that gradient- gradient-based attacks can get stuck in bad local minima (e.g., characterized by flat regions), where no adversarial example is found (F, in Fig. 3 When this failure is present, un- constrained attacks (i.e., attacks with - oo in Eq. 2 are also expected to fail, even if adversarial examples exist for sure in this setting (since the feasible domain includes samples from all classes). M5: Change Loss (Bad Local Minimum). We argue that this failure can be fixed by modifying the
5

---

loss function optimized by the attack, to avoid attack paths that lead to bad local minima. However, similarly to Ms. this is not trivial issue to fix, and is definitely not easy to automate. 1s: Unconstrained Attack Failure. Despite being difficult to find a suitable mitigation for F6. detecting it straightforward. To this end, we run an unconstrained attack on the given input æ, and we set (mr) = 1 if such an attack fails (and ( otherwise).
2.3 How to Use Our Framework
We summarize here the required steps that developers and practitioners should follow to debug their robustness evaluations with our IoAF.
Initialize Evaluation. The evaluation is initialized with chosen attack, along with its hyperparameters, number of samples, and the model evaluate.
Mitigate Loss-landscape Failures. Before computing the attack, the evaluation should get rid of loss-landscape failures, by computing Ij and Iz, and applying M, and M₂ accordingly. These indicators are computed on a small random subset of N samples, as the presence of either F2 or F2 would render useless the execution of the attack on all points, Both F, and
Fz are reported if they are detected on at least one sample in the given subset.
Run the Attack. If F1 and F2 are absent or have been fixed, the attack can be run on all samples.
Mitigate Attack Attack-specific Failures. Once the attack completes its execution, indicators 4 I, and Is should be checked. Depending on their output, M,, Ma, and M5 are applied, and the attack is repeated (on the affected samples). If no failures are found, but the attack is still failing, the value ls should be checked, and M, applied needed. As 1 and I2, ls is also computed on a subset of N samples to avoid unnecessary computations. The failure is reported if lo- = 1 for at least one sample.
Let us finally remark that when indicator triggers, one should not conclude that robustness may not be worsened by a different, more powerful attack. This is indeed a problem of any empirical evaluation, including software testing- which does not ensure that software is bug-free, but only that some known issues get fixed. Our methodology, similarly, highlights the presence of known failures in adversarial robustness evaluations and suggests how mitigate them with systematic protocol, taking a first concrete step toward making robustness evaluations more systematic and reliable.
3 Experiments
In this section, we demonstrate the effectiveness of our approach by re-evaluating the robustness of 7 defenses, and using our IoAF framework fix them. Additionally, we evaluate 6 more robust image classification models from a widely-used benchmark, discovering that their robustness claims might be unreliable. We also conduct the following additional experiments, reported in the appendix: (i) we describe how we set the thresholds for 1N and p. for I, in A.2; and (ii) we offer evidence that loAF can encompass perturbation models beyond ep norms by analyzing the robustness evaluations of one Windows malware detector in A.3, one Android malware detector in A.4, and one audio keyword-spotting model in A.4.
Experimental Setup. We run our attacks on an Intel x CPU E5-2670 v3, with 48 cores, 128 GB of RAM, equipped with Nvidia Quadro M6000 with 24 GB of memory, and we leverage the SecMI. library (32] to implement our methodology. We show failures of gradient-based attacks on 7
21 47), by inspecting them with our IoAF, are characterized by wrong robustness evaluations. We choose these defenses because is known that their original evaluations are incorrect 2 our goal here is to show that loAF would have identified and mitigated such errors. We include the evaluation of an adversarial detector 38 to highlight that IoAF is able to detect possible failures of this family of classifiers. We include the robustness evaluations of a Wide-ResNet [47] and an adversarially-Ee trained model [21], to highlight that our indicators do not trigger when they inspect sound evaluations. We defer details of the selected original evaluations to A. Lastly, to demonstrate that automated attacks do not provide guarantees on performing correct evaluations (without human intervention), we apply the AutoPGD attack [13] with Cross-Entropy loss (APGDCE) and the Diffeerence of Logits Ratio
previously published defenses [6
22
29
43
28
45
38
and most of them [16, [22 29
43
28
45

---

Table Indicator values (cols.) computed on the selected models (rows) using different attacks. The robust accuracy (RA) is reported in the last column (best values in bold). The symbol "Tr" denotes a transfer attack. The represents the detection of specific failure. We report in parentheses fraction of samples for which indicators 12, la. 14, and Is are active.
Model
Attack
1 1
13
14
15
18
RA
ST
PGD
0.00
ADV-T
PGD
0.48
Original
(100/1)
0.95
APGDeE
V (10/10)
0.99
DIST
APGDDLE
0.00
Patched
0.01
Original
V (10/10)
(23%)
(11%)
(4/10)
0.67
APGDcE
V (10/10)
(21%)
(2/11)
0.35
k-WTA
APGD DLR
(110/10)
(44110)
0.28
Patched
(6%)
(2/11)
0.09
Original
(110/10)
(33%)
(90%)
0.32
APGDcE
v /11//10
(3%)
(2/10)
0.12
IT
APGDDLA DLR
((10/10)
(5%)
((1110)
0.12
Patched
(3%)
0.00
Original
(100%
(3/10)
0.48
APGDCE
0.00
EN-DV
APGDDLK
0.00
Patched
0.00
Original
(74%%
(6%%
(8/10)
0.77
APGDCE
0.03
TWS
APGDDLR
((7/10)
0.68
Patched
(1%%
0.01
Original
0.85
Original (Tr)
V (78%)
0.74
JPEG-C
APGDGE (Tr)
(62%)
0.51
APGDDLR (Tr)
(85%)
0.70
Patched
0.01
Original
v (100%)
(10//10)
0.75
APGDcE
/ (3%)
0.02
DNR
APGDDLR
((/110)
0.20
Patched
((%%
0.03
(APGDDLR). Robustness is computed with the robust accuracy (RA) metric, quantified the ratio of samples classified correctly within given perturbation bound c. For I,, 12, and I, we set N = 10, 100, and the radius of the l2 ball r = 10-3 3 to match the step size a of the evaluations. The thresholds 7 of I2 and of 1, are evaluations are performed on 100 samples from the test dataset of the considered model, and for each attack we evaluate the robust accuracy with 8/255 for CIFAR models, and 0.5 for the MNIST ones. The step size a is set to match the original evaluations (as detailed in A.
and for 4 set 10. For I2, we the number of sampled neighboors
10% and 1% respectively (details about their calibration in A.2). All the robustness
Identifying and Fixing Attack Failures. We now delve into the description of the considered evaluations, which failures we detect and how we mitigate them, reporting the results in Table !
Correct Evaluations (ST, ADV-T). We first evaluate the robustness analysis of the Wide-ResNet and the adversarially-trained model, by applying PGD with CE loss, with n = 100 (number steps) and 0.03 (step size). Since no gradient obfuscation techniques have been used, the loss landscape indicators not trigger. Also, since the attacks smoothly converge, these evaluations not trigger any attack optimization indicator. Their robust accuracy is, respectively, 0% and 48%.

---

Defensive Distillation (DIST). Papernot et al. 29 use distillation to train classifier to saturate the last layer of the network, making the computations of gradients numerically unstable and impossible to calculate, triggering the I indicator, but also ls indicator as the optimizer can not explore the space. To patch this evaluation, we apply M, to overcome the numerical instability caused by DIST, forcing the attack to leverage the logits of the model instead of computing the softmax. Such fix reduces the robust accuracy from 95% to 1%. In contrast, DPGDDLR manages to decrease the robust accuracy to 0%, as it avoids the saturation issue by considering only the logits of the model, while APGD triggers the same issues of the original evaluation.
k- Winners Take All (k- WTA). Xiao et al. [43 develop classifier with a very noisy loss landscape, that destroys the meaningfulness of the directions of computed gradients. The original evaluation triggers 12. since the loss landscape is characterized by frequent fluctuations, but also Z5. as the applied PGD 225 returns the last point of the attack discarding adversarial examples found within the path. Moreover, due to the noise, the attack is not always reaching convergence, as signaled by 4,, also performing little exploration of the space, as flagged by Ig. Hence, we apply M2. performing EoT on the PGD loss, sampling 2000 points from a Normal distribution (0, I²1), with 8/255. We fix the implementation with M,, and increase the iterations with M4, thus reducing the robust accuracy from 67% to 9%. However, our evaluation still triggers 44 and Is. implying that for some points the result can be improved further by increasing the number of steps or the smoothing parameter y. Both APGDcE and APGDDLR trigger 12, since they are not applying EoT, and they partially activate I6. similarly to the original evaluation. Interestingly, APGDDLR converges better than APGDCE, as the latter triggers 44. Both attacks decrease the robust accuracy to 35% and 28%, a worst estimate than the one computed with the patched attack that directly handles the presence of the gradient obfuscation.
Imput Transformations (IT). Guo et al. 122 apply random affine input transformations to input images, producing a noisy loss landscape that varies at each prediction, thus its original robustness evaluation 12. Iz. and 14 indicators, for the same reasons of k- KAA Again, we apply M2, M, and Ma. using EoT with 7 = 200 and a = 8/255, decreasing the robust accuracy from 32% to 0%. Still, for some points (even if adversarial) the objective could still be improved, as 14 is active for them. Both APGD and APGDDLR are able to slightly decrease the robust accuracy to 12%, but they both trigger the /z and 1o indicator since they not addressing the high variability of the loss landscape.
Ensemble Diversity (EN-DV). Pang et al. (28 propose an ensemble model, evaluated with only 10 steps of PGD, thus triggers the Is indicator since the attack is not reaching convergence. As consequence, also the unbounded attack is failing, thus triggering Is. We apply the M,, mitigation and we set 100, already done by previous work [4]). This fix changes the robust accuracy from 48% to 0%. Not surprisingly, both APGDCE and APGDDIR are able to bring the robust accuracy to 0%, since they both automatically modulate the step size.
Turning a Weakness into a Strength (TWS). Yu et al. (55 propose an adversarial example defense, composed of different detectors. For simplicity, we only consider one of these, and we repeat their original evaluation with PGD without the normalization operation on the computed gradients, and = 50. This attack does not converge, triggering 44, and if fails to navigate the loss landscape due to the very small gradients, triggering also 4.. We apply M,, by setting n = 100, and we use the original implementation of PGD (with the normalization step), hence applying also Ms, achieving a drop in the robust accuracy from 77% 1%. For only one point, our evaluation triggers 4,, as better convergence could be achieved. Moving forward, APGDcE finds a robust accuracy of 3%, as is modulating the step size along with using the normalization of the gradient. On the other hand, APGDDLR triggers the I indicator and the robust accuracy remains at 68%, as the attack is not able to avoid the rejection applicd by the model.
JPEG Compression (JPEG-C). Das et al. [6] pre-process all input samples using JPEG compression, before feeding the input to the undefended network. We apply this defense on top of model with a reverse sigmoid layer [23], as was done in [44). The authors firstly state that they can not directly attack the defense due to its non-differentiabiliyy (thus triggering 4,), then they apply the DeepFool [27] attack against the undefended model. This attack has low transferability, as it is not maximizing the misclassification confidence [18]. The robustness evaluation triggers Is since the attack computed on the undefended model is not successful on the real defense. To patch this evaluation, we apply M,, approximating the propapropng of the JPEG compression and
the reverse sigmoid layers with BPDA [41). and we apply Ms by using PGD to maximize the misclassification confidence of the model. This fix is sufficient to reduce its robust accuracy from
8

---

Table 2: Indicator values (cols.) computed on the robust models (rows), using the APGD and APGDDI. attacks [13]. The robust accuracy (RA) is reported in the last column.
Model
Attack
11
12
11
14
15
4.
RA
APGD
V (10/10)
0.90
Stutz et al. 139
APGD JPGDDLR DLR
v (10/10)
0.90
APGD CE
v (4/10)
0.59
Carmon et al. 122
APGD DLR
(33110)
0.55
APGD CE
v (4/10)
0.62
Sehwag et al. 137
APGDDLR
(4/10)
0.57
APGD
(4/10)
0.62
Wu et al. 421
APGDDLR
v (3/10)
0.59
APGD
(2/10)
0.47
Ding et al. 20
APGD
(2/10)
0.49
APGD CE
v (4/10)
0.64
Rebuffi al.
35
APGDDLR
v (5/10)
0.65
74% 0.01%. Since APGD can not attack non-differemtiable models, run the attack against the undefended target and we transfer the adversarial examples on the defense. However, both of them trigger Is transfer attacks are not effective, keeping the robust accuracy to 51% and 70%.
Deep Neural Rejection (DNR). Sotgiu er al. [38 propose an adversarial detector encoding it as an additional class that captures the presence of attacks. Similarly to TWS, it was evaluated with PGD with no normalization of the norms of gradients. The attack does not converge, as it triggers 4, but also sometimes gets trapped inside the rejection class, triggering 4. Hence, we apply both M, and Ms, by increasing the number of iterations and considering an attack that avoids minimizing the score of the rejection class. These fixes reduce the robust accuracy from 75% to 3%. Interestingly, only APGD able to reduce the robust accuracy to 2%, also noting that it might be still beaten since is triggering 14. On the contrary, APGDDLR is reducing the robust accuracy only to 20%, by also triggering J.. By manually inspecting the outputs the model, we found that its scores tend to have all the same values, due to the SVM-RBF kernel that assigns low prediction outputs samples that are outside the distributions of the training samples. This causes the DLR loss to saturate due to the denominator becoming extremely small, making this attack weak against the DNR defense.
Evaluation of Robust Image Classification Models. We evaluate defenses recently published on top-tier venues, available through RobustBench [12 37 42 20 35 or their official repository [3), by applying APGD Cr and APGDDLn. using an lo perturbation bounded by 6 = 8/255. We report the results in Table Interestingly, we found that all these evaluations are unreliable, they trigger the Ia indicator. Hence, even without bounds on the perturbation model, the optimizer is unable to reach regions where adversarial examples are found. This suggests that applying Mo. i.c. making the attacks adaptive, can improve the trustworthiness of these robustness evaluations.
4 Related Work
Robustness Evaluations. Prior work focused on re-evaluating ftready-publisher defenses 9 41 showing that their adversarial robustness was significantly overestimated. The authors then suggested qualitative guidelines and best practices to avoid repeating the same evaluation mistakes in the future. However. the application of such guidelines remained mostly neglected, due to the inherent difficulty of applying them in an automated and systematic manner. To overcome this issue, in this work we have provided a systematic approach to debug adversarial robustness evaluations via the computation of the IoAF, which identify and characterize the main causes of failure of gradient- cradient-based attacks known to date. In addition, we have provided a semi-automated protocol that suggests how to apply the corresponding mitigations in the right order, whenever possible.
Benchmarks. Instead re-evaluating robustness of preiouuly-proppon defenses with an ad-hoc approach, the aforementioned papers do, some more systematic benchmarking approaches have

---

been proposed. Ling et al. [24] have proposed DEEPSEC, benchmark that tests several attacks against wide range of defenses. However, this framework was shown be flawed by several implementation issues and problems in the configuration of the attacks 77] Croce et al. [14] have proposed RobustBench, a benchmark that accepts state-of-the-art models as submissions, and tests them automatically with AutoAttack [13]. The same authors have also recently introduced some textual warnings when executing Autak, which are displayed if the strategy encounters specific issues (e.g., randomness and zero gradients). taking inspiration from what we have done more systematically in this paper. Yao et al. 44 have proposed an approach referred to as Adaptive AutoAnack, which automatically tests variety of attacks. by applying different configurations of parameters and objective functions. This approach has been shown to outperform AutoAttack by some margin, but at the expense of being much more computationally demanding.
While all these approaches are able to automatically evaluate adversarial robustness, they mostly blindly apply different attacks to show that the robust accuracy of some models can be decreased by a small fraction. None of the proposed benchmarks has implemented any mechanisms to detect known failures of gradient-based attacks and use them to patch the evaluations, except for the aforementioned warnings present in AutoAttack, and a recent flag added to RobustBench. This flag reports a potentially flawed evaluation when the black-box (square) attack finds lower robust accuracy values than the gradient- based attacks APGDcE. APGDDLR. and FAB. However, this not guaranteed to correctly detect all flawed evaluations, finding adversarial examples with black-box attacks is even more complicated and computationally demanding. Our approach finds the same issues without running any computationally -emaanding black-box attack, but just inspecting the failures in the optimization process of gradient-based attacks. For these reasons, we firmly believe that our approach based on the loAF will be extremely useful not only to dehug in more systematic and automated manner future robustness evaluations, but also to improve the current benchmarks.
5 Conclusions and Future Work
In this work, we have proposed the first automated testing approach that helps debug adversarial robustness evaluations, by characterizing and quantifying four known and two novel causes of failures of gradient- attacks. To this end, we have developed six Indicators of Attack Failure (IoAF), fixes. We have demonstrated the
and connected them with the corresponding effectiveness of our methodology by analyzing more than 15 models on three different domains, showing that most of the reported evaluations were flawed, and how to fix them with the systematic protocol we have defined, An interesting direction for future work would be to define mitigations that can be applied in a fully-automated way, including how to make attacks adaptive. Still, our work provides a set of debugging tools and systematic procedures that aim to sigmificantly reduce the number of manual interventions and the required skills to detect and fix flawed robustness evaluations.
To conclude, we firmly believe that integrating our work in current attack libraries and benchmarks will help avoid factual mistakes in adversarial robustness evaluations which have substantially hindered the development of effective defense mechanisms against adversarial examples thus far.
Acknowledgements
This work has been partly supported by the PRIN 2017 project RexLearn (grant no. 2017TWNMH2), funded by the Italian Ministry of Education, University and Research; and by BMK, BMDW, and the Province of Upper Austria in the frame of the COMET programme managed by FFG the COMET module S3AI; by the TESTABLE project, funded by the European Union's Horizon 2020 research and innovation programme (grant no. 101019206); and by the ELSA project, funded by the European Union's 's Horizon Europe research and innovation programme (grant 101070617).
References
[1] D. Arp, M. Spreitzenbarth, M. Hübner, H. Gascon, and K. Rieck. Drebin: Efficient and explainable detection of android malware in your pocket. In Proc. 21 Annual Network & Distributed System Security Symposium (NDSS). The Internet Society, 2014.
10

---

[2] A. Athalye, N. Carlini, and D. A. Wagner. Obfuscated gradients give a false sense of security: Cireumventing defenses adversarial examples. In ICML, volume 80 of JMLR Workshop and Conference Proceedings, pages 273-283. JML..org, 2018.
[3] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In ICLR, 2018.
[4] B. Biggio, I. Corona, D. Maiorea, B. Nelson, N. Šmdić, P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In H. Blockeel, K. Kersting, S. Nijssen, and F. Zelezný, editors, Machine Learning and Knowledge Discovery in Databases (ECML PKDD), Part III, volume 8190 of LNCS, pages 387-402. Springer Berlin Heidelberg, 2013.
[5] W. Brendel, J. Rauber, M. Kümmerer, I. Ustyuzhaninov, and M. Bethge. Accurate, reliable and fast robustness evaluation, 2019.
[6] Buckman, A. Roy, C. Raffel, and I. Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations, 2018.
[7] N. Carlini. A critique of the deepsec platform for security analysis of deep learning models, 2019.
[8] N. Carlini and D. Wagner. Defensive distillation not robust to adversarial examples, 2016.
[9] N. Carlini and D. A. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In B. M. Thuraisingham, B. Biggio, D. M. Freeman, B. Miller, and A. Sinha, editors, 10th ACM Workshop on Artificial Intelligence and Security, AlSec '17, pages 3-14, New York, NY, USA, 2017. ACM.
[10] N. Carlini and D. A. Wagner. Towards evaluating the robustness of neural networks, In IEEE Symposium on Security and Privacy, pages 39-57. IEEE Computer Society, 2017.
[11] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and Kurakin. On evaluating adversarial robustness, 2019.
[12] Y. Carmon, A. Raghunathan, L. Schmidt, J. C. Duchi, and Liang. Unlabeled data improves adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
[13] F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fre free attacks. In ICML, 2020.
[14] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein. Robustbench: standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL htttpp:///penrvveie www..wwww... nwt//foummiddd-eeeeoeoo (SKddSSKKPPCCCC
[15] Croce, S. Gowal, T. Brunner, E. Shelhamer, M. Hein, and T. Cemgil. Evaluating the adversarial robustness of adaptive test-time defenses. arXiv preprint arXiv:2202. 2022.
[16] N. Das. M. Shanbhogue, S.-T. Chen, F. Hohman, S. Li, L. Chen, M. E. Kounavis, and D. H. Chau. Shield: Fast, practical defense and vaccination for deep learning using jpeg compression. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 196- 204, 2018
[17] L. Demetrio, S. E. Coull, B. Biggio, G. Lagorio, A. Armando, and F. Roli. Adversarial exemples: A survey and experimental evaluation of practical attacks machine learning for windows malware detection. ACM Transactions on Privacy and Security (TOPS), 24(4):1-31, 2021.
[18] A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, A. Oprea, C. Nita-Rotaru, and Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In 28th USENIX security symposium (USENIX security 19), pages 321-338, 2019.
[19] A. Demontis, M. Melis, B. Biggio, D. Maiorca, D. Arp, K. Rieck, I. Corona, G. Giacinto, and Roli. Yes, machine learning can be more secure! case study on android malware detection. IEEE Trans. Dependable and Secure Computing, In press.

---

[20] G. W. Ding, Y. Sharma, K. Y. C. Lui, and R. Huang. Mma training: Direct input space margin maximization through adversarial training. In International Conference on Learning Representations, 2019.
[21] L. Engstrom, A. Ilyas, H. Salman, S. Santurkar, and D. Tsipras. Robustness (python library), 2019. URL BtRthttps:// //gp:::///hhuu com///adrrllal//bbsuseeess
[22] C. Guo, M. Rana, M. Cisse, and L. van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations, 2018.
[23] T. Lee, B. Edwards, I. Molloy, and D. Su. Defending against neural network model stealing attacks using deceptive perturbations. In 2019 IEEE Security and Privacy Workshops (SPW), pages 43-49, 2019. doi: 10. 10.1109/SP.2219.0000
[24] X. Ling, S. Ji, J. Zou, J. Wang. C. Wu, B. Li, and T. Wang. Deepsee: A uniform platform for security analysis of deep learning model. In 2019 IEEE Symposium on Security and Privacy (SP), pages 673-690, 2019. doi: 10.1 20.11099/..019.00022. 10..0003......
[25] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[26] D. Meng and H. Chen. MagNet: a two-pronged defense against adversarial examples. In 24th ACM Conf. Computer and Comm. Sec. (CCS), 2017.
[27] S.-M. Moosavi-Dezfooli, A. Fawzi, and Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 2574-2582, 2016.
[28] T. Pang, K. Xu, C. Du, N. Chen, and J. Zhu. Improving adversarial robustness via promoting ensemble diversity. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1970-4979) PMLR, 09-15 Jun 2019. URL httt:://proceedings. mlr press/v91 pxess/v97/pang19.. html
[29] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pages 582-597, May 2016. doi: 10.1109/SP.201.441.
[30] N. Papernot, P. D. McDaniel, and I. J. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. ArXiv e-prints, fess/60057777, 2016.
[31] N. Perraudin, P. Balazs, and P. L. Søndergaard. A fast griffin-lim algorithm. In 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pages 1 1-4. IEEE, 2013.
[32] M. Pintor, L. Demetrio, A. Sotgiu, M. Melis, A. Demontis, and B. Biggio. secml: Secure and explainable machine learning in python. SoftwareX, 18:101095, 2022. ISSN 2352-7110. doi: https:://do.org// wt. http://oorro.......oooooooo...2. 101095. http:tss::///// /ww. www.ssceiiiiiicieeeeeeeeeeeeeeee com/ ocieacc//arrucleepp////2271///10
[33] N. Popovic, D. P. Paudel, T. Probst, and L. Van Gool. Gradient obfuscation checklist test gives a false sense of security.
[34] E. Raff, J. Barker, J. Sylvester, R. Brandon, B. Catanzaro, and C. K. Nicholas. Malware detection by cating whole EXE, In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[35] S.-A. Rebuffi S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. A. Mann. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 34:29935-29999 2021.
[36] K. Roth, Y. Kilcher, and T. Hofmann. The odds are odd: A statistical test for detecting adversarial examples. International Conference on Machine Learning, pages 5498-5507. PMLR, 2019.
12

---

[37] V. Sehwag, S. Wang, P. Mittal, and S. Jana. Hydra: Pruning adversarially robust neural networks. Advances Neural Information Processing Systems (NeurIPS). 7. 2020.
[38] A. Sotgiu, A. Demontis, M. Melis, B. Biggio, G. Fumera, X. Feng, and F. Roli. Deep neural rejection against adversarial examples. EURASIP Journal on Information Security, 2020(1): 1-10, 2020.
[39] D. Stutz, M. Hein, and B. Schiele. Confidence-calibrated adversarial training: Generalizing to unseen attacks. In International Conference Machine Learning, pages 9155-9166. PMLR, 2020.
[40] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intrigu- ing properties of neural networks. In International Conference on Learning Representations, 2014. URL URLhttt://arrtt
org/ aat/a/as/11111 6199
[41] F. Tramer, N. Carlini, W. Brendel, and A. Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 33, 2020.
[42] D. Wu, S.-T. Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958-29909 2020.
[43] C. Xiao, P. Zhong, and C. Zheng. Resisting adversarial attacks by k-winners-take-all. 2020.
[44] C. Yao, Bielik, Tsankov, and M. Vechev. Automated discovery of adaptive attacks on adversarial defenses. Advances in Neural Information Processing Systems, 34, 2021.
[45] T. Yu, S. Hu, C. Guo, W. Chao, and K. Weinberger. A new defense against adversarial images: Turning a weakness into strength. Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Oct. 2019.
[46] X. Yuan, P. He, Q. Zhu, and X. Li. Adversarial examples: Attacks and defenses for deep learning. IEEE Transactions on Neural Networks and Learning Systems, 30(9):2805-288 -2224, 2019. doi: 10. 109/TNNLS 2018.2886011
[47] S. Zagoruyko and N. Komodakis. Wide residual networks. In E. R. H. Richard C. Wilson and W. A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C.30.87, URL
https://dx doi. org/ 5244/
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] We discuss the limitations in
Sect. 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In Sect. [ we specified that the aim of our work is not to break defenses in an harmful way. Our purpose only to help researchers improving their security evaluation.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] The code
will be submitted as supplementary material, and the instructions for reproducing the experiments are described in Sect. [
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] We describe the experimental protocol in Sect. g
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] The resourced used for the experiments are listed in Sect. 3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cited all the existing assets used for the experiments.
(b) Did you mention the license of the assets? [Yes] We cited the authors of the assets, and we provide the list of external assets along with the code.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We provide the code for computing the metrics as supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] All the assets we used are publicly available. (e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "yG3GowhJrZ", "reviewer": "Reviewer_xRJ5", "summary": "The paper touches on the problem of evaluating the robustness of models to adversarial examples. The proposed systematic evaluation protocol consists of 6 quantitative indicators of failure (IoFA) and corresponding fixes to address the problems. Gradient-based attack failures (e.g., shattered gradients) are described first and then indicators (e.g., unavailable gradients) are introduced accordingly. From this, methods (e.g., use BPDA) to improve the reliability of robustness evaluation are specified to mitigate loss landscape failures and attack optimization failures. Given positive experimental evidence, the proposed pipeline is claimed to be effective by inspecting 7 previously published defenses. Related work, limitations, and potential future work are discussed. Code and data will be public.", "questions": "1). Each failure and indicator is coupled with a solution; however, these solutions sound not trivial and automatic. How to decide on the hyper-parameters or settings to apply the suggested skills?\n\n2). Is there an example showing the before-and-after to underline the function of the proposed pipeline?\n\n3). The information in Figure 2 is minimal. It looks like a repetition of the description in the body. Also, Figure 1 is not clear enough for me to distinguish the difference between gray against black and red dot against a cross.\n\n4). The paper ends with the main experiments. Although it already takes huge efforts to involve 7 defenses, the work is not complete to me. Followed studies are necessary.\n", "limitations": "Limitations are discussed in Section 6.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1). The paper focuses on the valuable problem of evaluating and improving adversarial defense, which is critical to the robustness of modern deep learning methods in practice. 2). Each indicator is well-motivated by observed failures and is coupled with suggested fixes. 3). The introduced pipeline is successfully examined to improve existing defense baselines by thoughtful experiments.\n\nWeaknesses:\n1). Technically speaking, the contribution of this work is incremental. The proposed pipeline is not that impressive or novel; rather, it seems to be a pack of tricks to improve defense evaluation. 2). Although IoFA is well supported by cited works and described failures, its introduction lacks practical cases, where Figures 1 and 2 do not provide example failures and thus do not lead to a better understanding. 3). The reported experimental results appear to evidence the proposed methods, while there is a missing regarding the case analysis and further studies.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)", "Privacy and Security (e.g., consent)"]}, "extra_text": {}}, {"review_id": "q6b-v6toYdq", "reviewer": "Reviewer_tKpS", "summary": "This paper proposes a set of indicators of failure for gradient-based adversarial ML attacks, as well as a set of corresponding potential fixes, with the aim of improving current adversarial robustness evaluations through increasing automation and systematization of the process.\n\nThe argument is that current proposed checklists can give a false sense of security as they do not ensure a reliable evaluation. \n", "questions": "It seems that the indicators and mitigations in themselves are not novel. Is this the case?\n\nIn the results, the original technique is patched with the proposed mitigations. Was some consideration given to applying the same sort of patches to the APGD technique? (if that makes sense to do)\n\nA more general philosophical question is whether the changes to the attacks from the mitigations are better measuring robustness, as is claimed, or are simply different/improved attacks to work around the defences which stop the initial attacks? i.e. is this improving evaluation of robustness against known attacks or simply funding new attack techniques??\n\nMiscellaneous minor points\n- Equation (1) is a bit confusing as depending on whether y is the label or target, the approach is different. The footnote (2) on page 2 attempts to clarify this, but it is still a bit of a muddled exposition.\n- The discussion just above section 2.1 suggests Alg 1 should return the perturbation for which the objective function is minimised. Surely also some consideration should be given to perturbations which return a different classification than the base label – as this is usually the goal of the attack, and though correlated with the objective function value may be different.\n- Presumably the model parameters referred to in Algorithm 1 should include the model architecture as well, i.e. a surrogate model can be a distinct architecture? Or is this implicit already?\n- The text in section 2 talks of various options, including an objective being a combination of L and ||\\delta|| with some hyperparameters. Yet algorithm 1 (line 5) does not seem to accommodate this.\n", "limitations": "Limitations addressed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper brings together some reasonable indicators of different failure cases and some reasonable mitigations in each of those cases.\n\nMy main issues are\n-\tThe list of issues is almost certainly incomplete\n-\tThe indicators for each issue considered are not proven to guarantee to capture the issue (especially in cases where some case by case threshold is required)\n-\tThe mitigations are not guaranteed to be the only reasonable workaround.\n-\tThere are still hyperparameters to be set/fine-tuned empirically/ad hoc.\n-\t\nOverall, while bringing together these indicators and mitigations into as more systematic approach is admirable, there are no particular indicators or mitigations that are novel. All seem to have been considered before (in isolation, admittedly) or are just common sense. Thus this paper seems a reasonable guide for practical evaluation and best practise, or even a taxonomy,  but lacks novelty and impact.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lnthNxKjHbM", "reviewer": "Reviewer_xcQc", "summary": "This paper proposes a systematic protocol for improving adversarial example evaluations against common failures. The authors summarize six common failures and propose quantification metrics and meditations for each failure. Experiments on 7 previously broken defenses show that the proposed protocol could have identified and improved their adaptive attacks.", "questions": "My current score is based on the following understanding:\n* It is hard to prevent inadvertently weak attacks.\n* It is good and useful to have a systematic way of identifying weak attacks.\n* The proposed indicators provide a better understanding of how and why some attacks are weak.\n\nI am willing to raise my score if the following concerns are adequately addressed:\n* [Originality-Weaknesses-1/2, Significance-Weakness-1]\n  * Under each indicator, metric, and fix, clarify that they have been previously discussed in the literature.\n  * Please discuss the significance and technical contributions beyond previously known techniques.\n* Other weaknesses in the Quality section.", "limitations": "Limitations have been adequately discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\n\n**Strengths**\n* The systematic enforcement of failure checking and improving defense evaluation is novel.\n* The proposed protocol is a novel extension and identification of well-known pitfalls in evaluating adversarial example defenses.\n\n**Weaknesses**\n* **Most techniques are well-known [1, 30].** While I understand (and agree) that connecting well-known failures and fixes is valuable and proposing this systematic protocol is useful, I am concerned that I could not find many new insights from the current discussion. That is, the current discussion is more like a rigorous summary (in terms of a systematic protocol that is indeed novel) of well-known insights discussed in [1, 30]. Part of my concern is that these failures, indicators, and fixes are not clearly cited aside (by explicitly discussing that they have been previously explored).\n  * Failures 1-2 and their fixes are already sufficiently discussed by [1, 30]. Their indicators are also discussed by [11], e.g., through black-box attacks and explicitly checking randomness.\n  * Failures 3-6 and their fixes are already sufficiently discussed by [30].\n* **The evaluation is very similar to [1, 30].** The attacking procedures of many defenses are very similar to those provided by [1, 30], except for the discussion of indicators. For example:\n  * k-WTA follows a similar logic as in [30]\n  * IT and JPEG-C follow a similar logic as in [1]\n\n### Quality\n\n**Strengths**\n* The evaluation is sound and shows that proposed indicators effectively identify weak attacks and that optimizing towards the proposed metrics can improve the robustness evaluation.\n\n**Weaknesses**\n* **Unclear tightness of these indicators.** While I understand that passing these indicators does not mean a strong attack, it is suggested to briefly discuss how “tight” these quantifiable indicators are. That is, how likely a weak attacker (potentially outside the discussed defenses) could inadvertently optimize for these indicators yet result in a weak attack not detected by these metrics. I have these concerns because the hyper-parameters and thresholds are only decided by empirical results of a few defenses.\n* **Some robustness is still higher than in previous attacks.** I am curious why some evaluations show much higher robustness than previously attacked by [1, 30]. For example, the authors were not able to reduce the robustness of JPEG-C to 0%, as has been done by [1].\n* **On the visualization of evaluations.** At L12, the authors mentioned that the proposed indicators could be used to visualize evaluations. Is this claim missing from the paper? I am not sure if it refers to Figure 1. If so, this part is not clearly discussed.\n\n### Clarity\n\n**Strengths**\n* The presentation and visualization are generally good.\n\n**Weaknesses**\n* The evaluation part is dense, and the readers might need to jump forth and back to look for references for six indicators and fixes.\n\n### Significance\n\n**Strengths**\n* The idea of synthesizing known failures as indicators and providing quantifiable metrics can be useful.\n* Although most techniques are well-known, synthesizing them in a systematic protocol seems to be a better way of filtering out weak evaluations.\n\n**Weaknesses**\n* **Unclear significance in addition to known techniques.** I could not find many new insights from this paper, which is more like an execution manuscript for well-known adaptive attack techniques in a well-known way that has been previously discussed in [1, 30]. Given that most failures and fixes have already been sufficiently discussed in the literature, the only remaining significance seems to be proposing the six metrics. However, I am not sure if these metrics are significant enough, not to mention that some of them have also been discussed before.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a8ih1DygNlb", "reviewer": "Reviewer_VhaD", "summary": "The paper provides a set of “quantitative indicators” that can be used to determine if a given adversarial attack has been correctly implemented. By using such indicators, developers of “defenses” can assess whether the attack used to evaluate the defense is “flawed or correct”. The paper also experimentally demonstrates the application of its main proposal, showing that 7 existing defenses are \"potentially flawed\" due to the incorrect implementation of the corresponding attacks.\n\n**Personal comment**\n\nThe paper sets the bar very high by mentioning---many times and starting from the Abstract---the “false sense of security” provided by existing defenses against adversarial examples. Although I liked the initial tone, which intrigued me and induced me to read the rest of the paper with high expectations, I was disappointed by the overall “takeaways” provided by the paper. Indeed, my impression is that the actual contribution of the paper is just a piece of software that can be useful to “debug” evaluations of adversarial attacks---but in the single context of Computer Vision. The paper is full of descriptions and observations, but some of them are either obvious or (apparently) taken by prior work---preventing to determine the “true” contribution to the state of the art provided by the paper. Furthermore, some of the considered defenses have already been found to be \"flawed\", so I am skeptical about their corresponding \"false sense of security\" (which is still an overexxageration---discussed below).\n\nPut simply, I believe that the biggest problem of the paper is that it is trying too much. My general recommendation to the authors is to tone down the paper and focus on the specific contribution (which I think is there, but still hidden!) of their work to the state-of-the-art. \n", "questions": "*Incorrect Notation.* The notation of the paper is imprecise. For instance, Equation 1 is ill-defined:\n\n•\t“d” is not specified when defining “x”\n\n•\tWhat is “L”? It is only later determined that it is the “loss”, but there is no formal definition. \n\n•\tThe following statement is vague: “y ∈ {1, . . . , c} is either its label (for untargeted attacks) or the label of the target class (for targeted attacks)”. Actually, it makes no sense: if “y” is “its label” (i.e., the label of “x”) then why would it represent an “untargeted attack”? At most, it should be that “y is any label that is not the correct label of x”. \n\nReading the rest of Section 2 by assuming the definitions provided by the authors is confusing.\nThe authors should clearly explain what each term in Equation 1 means. If Eq.1 draws from past work, then the authors should clearly pinpoint which specific work is being used as basis (there are many variations of Eq.1 that have been proposed in the literature, e.g., [A], so it is crucial to pinpoint which work -if any- is being used as basis). \n\n\n*Only Computer Vision.* Although I understand that NeurIPS is a conference that primarily focuses on Computer Vision (CV) applications of Neural Networks (NN), the fact that the paper only considers attacks (and defenses) on CV is a significant drawback. Indeed, there are much more applications of NN than CV --- all of which have been found to be vulnerable to adversarial attacks (e.g., [B,D]), and all of which having “failures” that do not appear to be considered in this paper (e.g.,[C]); however, this paper only considers CV. What’s worse, is that the main proposal (i.e., the “IoAF”) seems exclusively tailored for the CV domain---which is epitomized by the decision of using SecML, which is a library that also is primarily suited for CV evaluations. In this context, the following statement is misleading: “We also include the evaluation of an adversarial detector [28] to highlight that our framework is able to detect possible failures of this family of classifiers.”: [28] is a “detector” of adversarial examples targeting CV models (indeed, the targeted classifier in [28] are trained on MNIST and CIFAR). To address this weakness, the authors should either (a) consider at least one application of NN which does not pertain to CV, and show that their proposal is effective at identifying failures (if any) also in such domain; or (b) tone down the paper by explicitly mentioning that all the contributions are restricted to CV. Note that the latter will tone down the paper, while the former will increase its impact significantly.\n\n\n*Exaggerated claims.* The paper states many times that “[the current situation] leads to a ‘false sense of security’”.  From a cybersecurity standpoint, this claim is overexaggerated: such “false sense of security” only exists if either (i) an attack is actually “common” in reality; or (ii) a defense is actually “deployed” in reality. If neither of the two is proven to be true, then the only consequence is that such false sense of security pertains solely to the research domain. I invite the authors to support such a statement with hard facts (justifying either the attack or the defense) or to significantly tone down such a statement.\nTO ADD TO THE ABOVE: it is surprising that the paper cites [10], but apparently does not consider a crucial statement of [10], namely: “A large body of work studies a threat model where the adversary is constrained to ℓp-bounded perturbations. This threat model is highly limited and does not perfectly match real-world threat.” Considering that Equation 1 (which is the “backbone” of the paper) clearly refers to lp bounded perturbations, it begs the question of what is the real-world utility of the contribution. Unfortunately, the remainder of Section 2 does not state whether the considered scenario is “restricted” to lp bounded perturbations or can also include completely unbounded perturbations; for instance, the “apply-constraints” function in Algorithm 1 is never defined (I even looked for it in the “supplementary material”). Perhaps the authors can elucidate more on this subject.\n\n\n*Where is the framework?* Let me quote the first of the claimed contributions reported in the Introduction: “we introduce a unified attack framework that captures the predominant styles of existing gradient-based attack methods, allowing us to identify six main failures that may arise during their optimization (Sect. 2);”. Where, exactly, in Section 2 is the “unified attack framework”? As explained above, the descriptions (algorithm and notation) of Section 2 are vague/unclear, and there are multiple concerns about the generalizability of the main proposal; then, the list of failures in Section 2.1 seems taken from existing literature (most notably [30]). This leads to the next weakness…\n\n*Unclear Contribution w.r.t. SotA?* It is not clear whether some of the “failures/indicators/mitigations” are (i) original content, (ii) reformulations of previous work, or (iii) completely taken from previous work. Sometimes, there are references provided to past work (e.g., F_2 cites [30]) but it is unclear whether such references are used to “support a statement\" (e.g., \"F_2 is a failure because [reason] which is based on the observations made in [30]”) or to “say that a given 'point' (e.g., a failure) is well-known\" (e.g., \"F_2 is explicitly stated in [30]\"). Note that such a weakness also spans over other parts of the paper. I strongly invite the authors to clearly differentiate original findings/intuitions from well-known “warnings” mentioned by past work. Furthermore, I emphasize that some of such “warnings” are obvious --- especially when considering the mitigations. Consider M3: “if I3 is true, then the implementation is flawed, so the implementation must be fixed accordingly” (but nothing more is said). That’s… common sense? Moreover, such a mitigation is hardly actionable.\n\n*Unsurprising results.* Most of the results are already well known, and hence the experiments, while being useful as a “demonstration”, provide little new insights. For instance, [22] (Distillation) is well known (since 2016) to be inaccurate (it only took a few months after the publication of the paper); it is almost surprising that the paper invalidating [22] is actually referenced in the manuscript (current citation [7]) but used in a completely different context (when explaining F2). Hence (to add to one of my previous weaknesses), what is the “false sense of security” provided by [22]? In this context, a much more \"impactful\" paper is [30], which thorouhgly elucidates novel issues with 13 defenses (which are much more than the 6 considered in this paper). I understand that the authors have a different objective than simply “invalidating prior defences”, but – as a reviewer and as a reader – the experimental evaluation is detrimental --- in its current form. I invite the authors to either (a) move the evaluation in a dedicated appendix, and then focus on strengthening the true contributions of the paper (which can be exclusively theoretical!); or (b) use the evaluation only to show the most original results (better if they are counterintuitive!).\n\n*Confusing organization.* I strongly believe the paper requires a re-organization that better elucidates the main proposal, due to the fragmentation of the “failures-indicators-mitigations”. Let me explain: the six failures are all listed in Section 2.1. Then, there is Section 3: for each of the six failures, an “indicator” of such failure is proposed; then, in subsection 3.1, there is a mitigation for each failure. Such an organization makes it difficult to “understand” the main proposal: listing all six failures, then all six indicators, and finally all six mitigations, can overwhelm a reader --- especially when it is not needed. Indeed, a quick look at Figure 2 shows that each failure is associated to exactly one indicator, which is then followed by exactly one mitigation. I would have understood the adoption of the current structure if some indicators were shared by more than one failure, or if some mitigations could have been related to more than one indicator/failure. However, considered the complete “1:1:1” mapping, I endorse the authors to better present their contribution: have each failure be followed by its indicator, and then by the corresponding mitigation. Such presentation is much more logical: while reading the paper I had to go back to the previous sections to recall the corresponding failure (or indicator). Potentially, such organization can be further strengthened by integrating a proof-of-concept experiment.\n\nSome lingering issues and suggestions:\n\n•\tIn Section 5: “our work […] offers a systematic methodology to provide better robustness evaluations”. The “systematic methodology” stands on very brittle foundations (see my main review) and desperately requires an adjustment. My (personal!) impression is that the paper is being oversold: I firmly believe that the contribution is a low-level software toolkit that can be useful in some cases, which is far from being a “systematic methodology”\n\n•\tA significant improvement would be listing “all” papers whose evaluations are flawed. Perhaps the search can be restricted only to the papers published in the last few years, and/or published in a few selected venues. Potentially, such analysis can also highlight which papers have been “already” determined to be “flawed” (by other works), and which are found to be “flawed” only by following the proposed set of indicators. Such an analysis would be a far stronger motivation and contribution, and induce almost a straight “accept” from my end.\n\n•\t*Where is the “perturbation”?* My intuition is that the adversarial perturbation is always introduced in the “feature space” (according to Equation 1, it is applied to “x” which is the input sample to the ML model). If this is the case, then the approach is not tailored for attacks carried out in the problem space, which are those resembling more realistic scenarios [C]. Of course, a possible response is that \"Our focus is on gradient-based attacks, which are typically crafted in the feature space\", to which I would reply that \"Such attacks have been indicated many times to be difficult to realize in reality\". In other words, the restricted focus of the paper is an intrinsic limitation for realisic and \"pragmatic\" takeaways.\n\n•\t*Flawed logic?* In the Introduction, the paper reports that “13 defenses […] were found to be ineffective due to the difficulty of optimizing the attacks correctly [30].” My intuition is as follows: the attacks used to “validate” those 13 defenses were not correctly implemented, hence the “success” (which I assume to be a lower “misclassification rate”) of such defenses were overestimated. If such a reasoning is correct, then it raises the question: the “attack” may not have been correctly implemented, but the misclassifications still appeared so it was (by definition) a “successful” attack---which the defense mitigated. Granted, it was a “different” attack than the one envisioned in the actual paper, but it still induced misclassifications (which were mitigated by the defense). In other words: substantiating the observation that “defenses are ineffective” by stating that “attacks were not correctly optimized” is logically flawed. The only takeaway that can be derived from that logic is that “the defense works against a ‘different’ attack, and the true efficacy of the defense against the ‘expected’ attack is not yet proven”. Indeed, the paper should focus on the implementation of the defense rather than of the attack. I truly invite the authors to clarify this “weakness” and prove me wrong (if I am). \n\n\nEXTERNAL REFERENCES\n[A]: \"Adversarial examples for malware detection.\" European symposium on research in computer security. Springer, 2017.\n\n[B]: \"Deep reinforcement adversarial learning against botnet evasion attacks.\" IEEE Transactions on Network and Service Management 17.4 (2020): 1975-1987.\n\n[C]: \"Intriguing properties of adversarial ml attacks in the problem space.\" 2020 IEEE symposium on security and privacy (SP). IEEE, 2020.\n\n[D]: \"Improving robustness of {ML} classifiers against realizable evasion attacks using conserved features.\" 28th USENIX Security Symposium (USENIX Security 19). 2019\n\n[E]: \"Are we learning yet? a meta review of evaluation failures across machine learning.\" Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2021.", "limitations": "An intrinsic (and not reported) limitation is that the work mostly focuses on attacks/defenses in the computer vision domain – neglecting the plethora of other domains in which adversarial examples can be conceived. \n\nAnother limitation (which is reported, but still significant and should be a subject of discussion here) is that the proposed set of indicators are only applicable “today”: if new “failures” arise (which are likely, considered the fast advances in this research field) then the current proposal will not be able to detect them --- hence potentially inducing the same sense of “false security” which the authors aim to remove. The issue, here, is that it is a \"devil's proof\": the contribution shows some indicators---but we cannot now if such indicators can cover all possible causes of \"failures\" that may affect current adversarial ML evaluations. \nPerhaps the authors could attempt to remedy to the abovementioend issue by \"promising\" to maintain the repository for several years, so that if new failures are found (affecting either current or future researches) then they will be integrated in the toolkit and usable by future researches.\n**My concern is this:** the moment a new paper finds a \"failure\" that is not included in those reported in this paper, then the (currently small) contribution of this paper to the state of the art will be insignificant.\n\n################\n\nUPDATE AFTER INITIAL REBUTTAL: the experiments on malware made me increase my score from 4 to 5, due to \"practically proving\" that the propose methodology can be adapted to cover also different domains than CV.\n\nUPDATE AFTER AUTHOR'S DISCUSSION: I am increasing the score to a 6 (I would rate it a 6.4)\n", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "HIGH LEVEL:\n\n**Originality:** Poor. The only true contribution, in my opinion, are the indicators---which are ~~limited to~~ evaluated on a single application domain (Computer Vision). Investigating \"failures\" in ML research is not new (e.g., [E]), and many works also do this in the adversarial ML domain (e.g. [30])\n\n**Clarity:** Good. In general, the paper reads well, the motivations are properly explained, and figures/tables are appropriate.\n\n**Quality:** Average. The notation is faulty and the organization can be improved to better reflect the true contribution of the paper.\n\n**Significance:** Poor. Most of the content is well known, and the ~~limited applicability~~ limited evaluation hinders its overall significance to the wide-spectrum of potential ML applications.\n\n\n\nLOW-LEVEL\n\n**Strenghts**:\n+ Some findings have practical utility for developers\n+ The implementation can be useful for future work\n\n\n**Weaknesses**\n- Incorrect notation\n- Only Computer Vision\n- Exaggerated claims\n- “Unsurprising” results\n- Where is the framework?\n- Unclear contribution w.r.t. SotA\n- Confusing organization\n\nREMARK\n\nI thank the authors for their paper: despite my criticism, I endorse their line of research and I would have little objections against this paper if it were submitted to a less prestigious venue.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept