Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper touches on the problem of evaluating the robustness of models to adversarial examples. The proposed systematic evaluation protocol consists of 6 quantitative indicators of failure (IoFA) and corresponding fixes to address the problems. Gradient-based attack failures (e.g., shattered gradients) are described first and then indicators (e.g., unavailable gradients) are introduced accordingly. From this, methods (e.g., use BPDA) to improve the reliability of robustness evaluation are specified to mitigate loss landscape failures and attack optimization failures. Given positive experimental evidence, the proposed pipeline is claimed to be effective by inspecting 7 previously published defenses. Related work, limitations, and potential future work are discussed. Code and data will be public.
strengths_and_weaknesses: Strengths:
1). The paper focuses on the valuable problem of evaluating and improving adversarial defense, which is critical to the robustness of modern deep learning methods in practice. 2). Each indicator is well-motivated by observed failures and is coupled with suggested fixes. 3). The introduced pipeline is successfully examined to improve existing defense baselines by thoughtful experiments.

Weaknesses:
1). Technically speaking, the contribution of this work is incremental. The proposed pipeline is not that impressive or novel; rather, it seems to be a pack of tricks to improve defense evaluation. 2). Although IoFA is well supported by cited works and described failures, its introduction lacks practical cases, where Figures 1 and 2 do not provide example failures and thus do not lead to a better understanding. 3). The reported experimental results appear to evidence the proposed methods, while there is a missing regarding the case analysis and further studies.

questions: 1). Each failure and indicator is coupled with a solution; however, these solutions sound not trivial and automatic. How to decide on the hyper-parameters or settings to apply the suggested skills?

2). Is there an example showing the before-and-after to underline the function of the proposed pipeline?

3). The information in Figure 2 is minimal. It looks like a repetition of the description in the body. Also, Figure 1 is not clear enough for me to distinguish the difference between gray against black and red dot against a cross.

4). The paper ends with the main experiments. Although it already takes huge efforts to involve 7 defenses, the work is not complete to me. Followed studies are necessary.

limitations: Limitations are discussed in Section 6.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a systematic protocol for improving adversarial example evaluations against common failures. The authors summarize six common failures and propose quantification metrics and meditations for each failure. Experiments on 7 previously broken defenses show that the proposed protocol could have identified and improved their adaptive attacks.
strengths_and_weaknesses: ### Originality

**Strengths**
* The systematic enforcement of failure checking and improving defense evaluation is novel.
* The proposed protocol is a novel extension and identification of well-known pitfalls in evaluating adversarial example defenses.

**Weaknesses**
* **Most techniques are well-known [1, 30].** While I understand (and agree) that connecting well-known failures and fixes is valuable and proposing this systematic protocol is useful, I am concerned that I could not find many new insights from the current discussion. That is, the current discussion is more like a rigorous summary (in terms of a systematic protocol that is indeed novel) of well-known insights discussed in [1, 30]. Part of my concern is that these failures, indicators, and fixes are not clearly cited aside (by explicitly discussing that they have been previously explored).
  * Failures 1-2 and their fixes are already sufficiently discussed by [1, 30]. Their indicators are also discussed by [11], e.g., through black-box attacks and explicitly checking randomness.
  * Failures 3-6 and their fixes are already sufficiently discussed by [30].
* **The evaluation is very similar to [1, 30].** The attacking procedures of many defenses are very similar to those provided by [1, 30], except for the discussion of indicators. For example:
  * k-WTA follows a similar logic as in [30]
  * IT and JPEG-C follow a similar logic as in [1]

### Quality

**Strengths**
* The evaluation is sound and shows that proposed indicators effectively identify weak attacks and that optimizing towards the proposed metrics can improve the robustness evaluation.

**Weaknesses**
* **Unclear tightness of these indicators.** While I understand that passing these indicators does not mean a strong attack, it is suggested to briefly discuss how “tight” these quantifiable indicators are. That is, how likely a weak attacker (potentially outside the discussed defenses) could inadvertently optimize for these indicators yet result in a weak attack not detected by these metrics. I have these concerns because the hyper-parameters and thresholds are only decided by empirical results of a few defenses.
* **Some robustness is still higher than in previous attacks.** I am curious why some evaluations show much higher robustness than previously attacked by [1, 30]. For example, the authors were not able to reduce the robustness of JPEG-C to 0%, as has been done by [1].
* **On the visualization of evaluations.** At L12, the authors mentioned that the proposed indicators could be used to visualize evaluations. Is this claim missing from the paper? I am not sure if it refers to Figure 1. If so, this part is not clearly discussed.

### Clarity

**Strengths**
* The presentation and visualization are generally good.

**Weaknesses**
* The evaluation part is dense, and the readers might need to jump forth and back to look for references for six indicators and fixes.

### Significance

**Strengths**
* The idea of synthesizing known failures as indicators and providing quantifiable metrics can be useful.
* Although most techniques are well-known, synthesizing them in a systematic protocol seems to be a better way of filtering out weak evaluations.

**Weaknesses**
* **Unclear significance in addition to known techniques.** I could not find many new insights from this paper, which is more like an execution manuscript for well-known adaptive attack techniques in a well-known way that has been previously discussed in [1, 30]. Given that most failures and fixes have already been sufficiently discussed in the literature, the only remaining significance seems to be proposing the six metrics. However, I am not sure if these metrics are significant enough, not to mention that some of them have also been discussed before.
questions: My current score is based on the following understanding:
* It is hard to prevent inadvertently weak attacks.
* It is good and useful to have a systematic way of identifying weak attacks.
* The proposed indicators provide a better understanding of how and why some attacks are weak.

I am willing to raise my score if the following concerns are adequately addressed:
* [Originality-Weaknesses-1/2, Significance-Weakness-1]
  * Under each indicator, metric, and fix, clarify that they have been previously discussed in the literature.
  * Please discuss the significance and technical contributions beyond previously known techniques.
* Other weaknesses in the Quality section.
limitations: Limitations have been adequately discussed in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a set of indicators of failure for gradient-based adversarial ML attacks, as well as a set of corresponding potential fixes, with the aim of improving current adversarial robustness evaluations through increasing automation and systematization of the process.

The argument is that current proposed checklists can give a false sense of security as they do not ensure a reliable evaluation. 

strengths_and_weaknesses: The paper brings together some reasonable indicators of different failure cases and some reasonable mitigations in each of those cases.

My main issues are
-	The list of issues is almost certainly incomplete
-	The indicators for each issue considered are not proven to guarantee to capture the issue (especially in cases where some case by case threshold is required)
-	The mitigations are not guaranteed to be the only reasonable workaround.
-	There are still hyperparameters to be set/fine-tuned empirically/ad hoc.
-	
Overall, while bringing together these indicators and mitigations into as more systematic approach is admirable, there are no particular indicators or mitigations that are novel. All seem to have been considered before (in isolation, admittedly) or are just common sense. Thus this paper seems a reasonable guide for practical evaluation and best practise, or even a taxonomy,  but lacks novelty and impact.

questions: It seems that the indicators and mitigations in themselves are not novel. Is this the case?

In the results, the original technique is patched with the proposed mitigations. Was some consideration given to applying the same sort of patches to the APGD technique? (if that makes sense to do)

A more general philosophical question is whether the changes to the attacks from the mitigations are better measuring robustness, as is claimed, or are simply different/improved attacks to work around the defences which stop the initial attacks? i.e. is this improving evaluation of robustness against known attacks or simply funding new attack techniques??

Miscellaneous minor points
- Equation (1) is a bit confusing as depending on whether y is the label or target, the approach is different. The footnote (2) on page 2 attempts to clarify this, but it is still a bit of a muddled exposition.
- The discussion just above section 2.1 suggests Alg 1 should return the perturbation for which the objective function is minimised. Surely also some consideration should be given to perturbations which return a different classification than the base label – as this is usually the goal of the attack, and though correlated with the objective function value may be different.
- Presumably the model parameters referred to in Algorithm 1 should include the model architecture as well, i.e. a surrogate model can be a distinct architecture? Or is this implicit already?
- The text in section 2 talks of various options, including an objective being a combination of L and ||\delta|| with some hyperparameters. Yet algorithm 1 (line 5) does not seem to accommodate this.

limitations: Limitations addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper provides a set of “quantitative indicators” that can be used to determine if a given adversarial attack has been correctly implemented. By using such indicators, developers of “defenses” can assess whether the attack used to evaluate the defense is “flawed or correct”. The paper also experimentally demonstrates the application of its main proposal, showing that 7 existing defenses are "potentially flawed" due to the incorrect implementation of the corresponding attacks.

**Personal comment**

The paper sets the bar very high by mentioning---many times and starting from the Abstract---the “false sense of security” provided by existing defenses against adversarial examples. Although I liked the initial tone, which intrigued me and induced me to read the rest of the paper with high expectations, I was disappointed by the overall “takeaways” provided by the paper. Indeed, my impression is that the actual contribution of the paper is just a piece of software that can be useful to “debug” evaluations of adversarial attacks---but in the single context of Computer Vision. The paper is full of descriptions and observations, but some of them are either obvious or (apparently) taken by prior work---preventing to determine the “true” contribution to the state of the art provided by the paper. Furthermore, some of the considered defenses have already been found to be "flawed", so I am skeptical about their corresponding "false sense of security" (which is still an overexxageration---discussed below).

Put simply, I believe that the biggest problem of the paper is that it is trying too much. My general recommendation to the authors is to tone down the paper and focus on the specific contribution (which I think is there, but still hidden!) of their work to the state-of-the-art. 

strengths_and_weaknesses: HIGH LEVEL:

**Originality:** Poor. The only true contribution, in my opinion, are the indicators---which are ~~limited to~~ evaluated on a single application domain (Computer Vision). Investigating "failures" in ML research is not new (e.g., [E]), and many works also do this in the adversarial ML domain (e.g. [30])

**Clarity:** Good. In general, the paper reads well, the motivations are properly explained, and figures/tables are appropriate.

**Quality:** Average. The notation is faulty and the organization can be improved to better reflect the true contribution of the paper.

**Significance:** Poor. Most of the content is well known, and the ~~limited applicability~~ limited evaluation hinders its overall significance to the wide-spectrum of potential ML applications.



LOW-LEVEL

**Strenghts**:
+ Some findings have practical utility for developers
+ The implementation can be useful for future work


**Weaknesses**
- Incorrect notation
- Only Computer Vision
- Exaggerated claims
- “Unsurprising” results
- Where is the framework?
- Unclear contribution w.r.t. SotA
- Confusing organization

REMARK

I thank the authors for their paper: despite my criticism, I endorse their line of research and I would have little objections against this paper if it were submitted to a less prestigious venue.

questions: *Incorrect Notation.* The notation of the paper is imprecise. For instance, Equation 1 is ill-defined:

•	“d” is not specified when defining “x”

•	What is “L”? It is only later determined that it is the “loss”, but there is no formal definition. 

•	The following statement is vague: “y ∈ {1, . . . , c} is either its label (for untargeted attacks) or the label of the target class (for targeted attacks)”. Actually, it makes no sense: if “y” is “its label” (i.e., the label of “x”) then why would it represent an “untargeted attack”? At most, it should be that “y is any label that is not the correct label of x”. 

Reading the rest of Section 2 by assuming the definitions provided by the authors is confusing.
The authors should clearly explain what each term in Equation 1 means. If Eq.1 draws from past work, then the authors should clearly pinpoint which specific work is being used as basis (there are many variations of Eq.1 that have been proposed in the literature, e.g., [A], so it is crucial to pinpoint which work -if any- is being used as basis). 


*Only Computer Vision.* Although I understand that NeurIPS is a conference that primarily focuses on Computer Vision (CV) applications of Neural Networks (NN), the fact that the paper only considers attacks (and defenses) on CV is a significant drawback. Indeed, there are much more applications of NN than CV --- all of which have been found to be vulnerable to adversarial attacks (e.g., [B,D]), and all of which having “failures” that do not appear to be considered in this paper (e.g.,[C]); however, this paper only considers CV. What’s worse, is that the main proposal (i.e., the “IoAF”) seems exclusively tailored for the CV domain---which is epitomized by the decision of using SecML, which is a library that also is primarily suited for CV evaluations. In this context, the following statement is misleading: “We also include the evaluation of an adversarial detector [28] to highlight that our framework is able to detect possible failures of this family of classifiers.”: [28] is a “detector” of adversarial examples targeting CV models (indeed, the targeted classifier in [28] are trained on MNIST and CIFAR). To address this weakness, the authors should either (a) consider at least one application of NN which does not pertain to CV, and show that their proposal is effective at identifying failures (if any) also in such domain; or (b) tone down the paper by explicitly mentioning that all the contributions are restricted to CV. Note that the latter will tone down the paper, while the former will increase its impact significantly.


*Exaggerated claims.* The paper states many times that “[the current situation] leads to a ‘false sense of security’”.  From a cybersecurity standpoint, this claim is overexaggerated: such “false sense of security” only exists if either (i) an attack is actually “common” in reality; or (ii) a defense is actually “deployed” in reality. If neither of the two is proven to be true, then the only consequence is that such false sense of security pertains solely to the research domain. I invite the authors to support such a statement with hard facts (justifying either the attack or the defense) or to significantly tone down such a statement.
TO ADD TO THE ABOVE: it is surprising that the paper cites [10], but apparently does not consider a crucial statement of [10], namely: “A large body of work studies a threat model where the adversary is constrained to ℓp-bounded perturbations. This threat model is highly limited and does not perfectly match real-world threat.” Considering that Equation 1 (which is the “backbone” of the paper) clearly refers to lp bounded perturbations, it begs the question of what is the real-world utility of the contribution. Unfortunately, the remainder of Section 2 does not state whether the considered scenario is “restricted” to lp bounded perturbations or can also include completely unbounded perturbations; for instance, the “apply-constraints” function in Algorithm 1 is never defined (I even looked for it in the “supplementary material”). Perhaps the authors can elucidate more on this subject.


*Where is the framework?* Let me quote the first of the claimed contributions reported in the Introduction: “we introduce a unified attack framework that captures the predominant styles of existing gradient-based attack methods, allowing us to identify six main failures that may arise during their optimization (Sect. 2);”. Where, exactly, in Section 2 is the “unified attack framework”? As explained above, the descriptions (algorithm and notation) of Section 2 are vague/unclear, and there are multiple concerns about the generalizability of the main proposal; then, the list of failures in Section 2.1 seems taken from existing literature (most notably [30]). This leads to the next weakness…

*Unclear Contribution w.r.t. SotA?* It is not clear whether some of the “failures/indicators/mitigations” are (i) original content, (ii) reformulations of previous work, or (iii) completely taken from previous work. Sometimes, there are references provided to past work (e.g., F_2 cites [30]) but it is unclear whether such references are used to “support a statement" (e.g., "F_2 is a failure because [reason] which is based on the observations made in [30]”) or to “say that a given 'point' (e.g., a failure) is well-known" (e.g., "F_2 is explicitly stated in [30]"). Note that such a weakness also spans over other parts of the paper. I strongly invite the authors to clearly differentiate original findings/intuitions from well-known “warnings” mentioned by past work. Furthermore, I emphasize that some of such “warnings” are obvious --- especially when considering the mitigations. Consider M3: “if I3 is true, then the implementation is flawed, so the implementation must be fixed accordingly” (but nothing more is said). That’s… common sense? Moreover, such a mitigation is hardly actionable.

*Unsurprising results.* Most of the results are already well known, and hence the experiments, while being useful as a “demonstration”, provide little new insights. For instance, [22] (Distillation) is well known (since 2016) to be inaccurate (it only took a few months after the publication of the paper); it is almost surprising that the paper invalidating [22] is actually referenced in the manuscript (current citation [7]) but used in a completely different context (when explaining F2). Hence (to add to one of my previous weaknesses), what is the “false sense of security” provided by [22]? In this context, a much more "impactful" paper is [30], which thorouhgly elucidates novel issues with 13 defenses (which are much more than the 6 considered in this paper). I understand that the authors have a different objective than simply “invalidating prior defences”, but – as a reviewer and as a reader – the experimental evaluation is detrimental --- in its current form. I invite the authors to either (a) move the evaluation in a dedicated appendix, and then focus on strengthening the true contributions of the paper (which can be exclusively theoretical!); or (b) use the evaluation only to show the most original results (better if they are counterintuitive!).

*Confusing organization.* I strongly believe the paper requires a re-organization that better elucidates the main proposal, due to the fragmentation of the “failures-indicators-mitigations”. Let me explain: the six failures are all listed in Section 2.1. Then, there is Section 3: for each of the six failures, an “indicator” of such failure is proposed; then, in subsection 3.1, there is a mitigation for each failure. Such an organization makes it difficult to “understand” the main proposal: listing all six failures, then all six indicators, and finally all six mitigations, can overwhelm a reader --- especially when it is not needed. Indeed, a quick look at Figure 2 shows that each failure is associated to exactly one indicator, which is then followed by exactly one mitigation. I would have understood the adoption of the current structure if some indicators were shared by more than one failure, or if some mitigations could have been related to more than one indicator/failure. However, considered the complete “1:1:1” mapping, I endorse the authors to better present their contribution: have each failure be followed by its indicator, and then by the corresponding mitigation. Such presentation is much more logical: while reading the paper I had to go back to the previous sections to recall the corresponding failure (or indicator). Potentially, such organization can be further strengthened by integrating a proof-of-concept experiment.

Some lingering issues and suggestions:

•	In Section 5: “our work […] offers a systematic methodology to provide better robustness evaluations”. The “systematic methodology” stands on very brittle foundations (see my main review) and desperately requires an adjustment. My (personal!) impression is that the paper is being oversold: I firmly believe that the contribution is a low-level software toolkit that can be useful in some cases, which is far from being a “systematic methodology”

•	A significant improvement would be listing “all” papers whose evaluations are flawed. Perhaps the search can be restricted only to the papers published in the last few years, and/or published in a few selected venues. Potentially, such analysis can also highlight which papers have been “already” determined to be “flawed” (by other works), and which are found to be “flawed” only by following the proposed set of indicators. Such an analysis would be a far stronger motivation and contribution, and induce almost a straight “accept” from my end.

•	*Where is the “perturbation”?* My intuition is that the adversarial perturbation is always introduced in the “feature space” (according to Equation 1, it is applied to “x” which is the input sample to the ML model). If this is the case, then the approach is not tailored for attacks carried out in the problem space, which are those resembling more realistic scenarios [C]. Of course, a possible response is that "Our focus is on gradient-based attacks, which are typically crafted in the feature space", to which I would reply that "Such attacks have been indicated many times to be difficult to realize in reality". In other words, the restricted focus of the paper is an intrinsic limitation for realisic and "pragmatic" takeaways.

•	*Flawed logic?* In the Introduction, the paper reports that “13 defenses […] were found to be ineffective due to the difficulty of optimizing the attacks correctly [30].” My intuition is as follows: the attacks used to “validate” those 13 defenses were not correctly implemented, hence the “success” (which I assume to be a lower “misclassification rate”) of such defenses were overestimated. If such a reasoning is correct, then it raises the question: the “attack” may not have been correctly implemented, but the misclassifications still appeared so it was (by definition) a “successful” attack---which the defense mitigated. Granted, it was a “different” attack than the one envisioned in the actual paper, but it still induced misclassifications (which were mitigated by the defense). In other words: substantiating the observation that “defenses are ineffective” by stating that “attacks were not correctly optimized” is logically flawed. The only takeaway that can be derived from that logic is that “the defense works against a ‘different’ attack, and the true efficacy of the defense against the ‘expected’ attack is not yet proven”. Indeed, the paper should focus on the implementation of the defense rather than of the attack. I truly invite the authors to clarify this “weakness” and prove me wrong (if I am). 


EXTERNAL REFERENCES
[A]: "Adversarial examples for malware detection." European symposium on research in computer security. Springer, 2017.

[B]: "Deep reinforcement adversarial learning against botnet evasion attacks." IEEE Transactions on Network and Service Management 17.4 (2020): 1975-1987.

[C]: "Intriguing properties of adversarial ml attacks in the problem space." 2020 IEEE symposium on security and privacy (SP). IEEE, 2020.

[D]: "Improving robustness of {ML} classifiers against realizable evasion attacks using conserved features." 28th USENIX Security Symposium (USENIX Security 19). 2019

[E]: "Are we learning yet? a meta review of evaluation failures across machine learning." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2021.
limitations: An intrinsic (and not reported) limitation is that the work mostly focuses on attacks/defenses in the computer vision domain – neglecting the plethora of other domains in which adversarial examples can be conceived. 

Another limitation (which is reported, but still significant and should be a subject of discussion here) is that the proposed set of indicators are only applicable “today”: if new “failures” arise (which are likely, considered the fast advances in this research field) then the current proposal will not be able to detect them --- hence potentially inducing the same sense of “false security” which the authors aim to remove. The issue, here, is that it is a "devil's proof": the contribution shows some indicators---but we cannot now if such indicators can cover all possible causes of "failures" that may affect current adversarial ML evaluations. 
Perhaps the authors could attempt to remedy to the abovementioend issue by "promising" to maintain the repository for several years, so that if new failures are found (affecting either current or future researches) then they will be integrated in the toolkit and usable by future researches.
**My concern is this:** the moment a new paper finds a "failure" that is not included in those reported in this paper, then the (currently small) contribution of this paper to the state of the art will be insignificant.

################

UPDATE AFTER INITIAL REBUTTAL: the experiments on malware made me increase my score from 4 to 5, due to "practically proving" that the propose methodology can be adapted to cover also different domains than CV.

UPDATE AFTER AUTHOR'S DISCUSSION: I am increasing the score to a 6 (I would rate it a 6.4)
review_title: null

Meta-review: metareview: First, thank you to the reviewers and authors for an in-depth discussion on the contributions and framing of this paper. There’s no doubt this paper was improved over the course of the rebuttal period. And thank you again to the reviewers for having participated in a discussion to clear up the final points of this paper.

This paper essentially presents a checklist of best practices for adversarial evaluation. Such a contribution would hopefully induce more rigorous evaluation standards around adversarial attack research, which is currently often stuck in this feedback loop of empirical attack and defences. New attacks are easily stopped by slight modifications to defences, but they fail under slightly modified attacks.  The authors propose to categorise various attack failures that plague prior work and propose a method to identify these failures ahead of time. They also suggest mitigations to fix these possible sources of failure. In essence, this paper is a combination of survey, reproduction, and opinion paper all in one.

**Strength**: The main strength of this paper is that the framework developed by the authors is practical. Researchers will be able to use this framework to _check_ the rigor of their empirical investigations, hopefully leading to an increase in standards field-wide. However, they also provide strong empirical results across three domains to validate their method of identifying failures.

**Weakness**: There is no “novel” technical contribution (i.e., no new attack & defence). The paper is more a re-hashing of prior approaches. I assume there will be plenty papers of the first variety (with impressive technical chops) at the conference, however, and think this paper stands on its own a different flavour whose content is worth acceptance.
confidence: Less certain
Decision: Accept