Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper design four quantitative indicators of attack failures (IoAFs) that might help to systemically automatically check the correctness of robustness evaluations. Besides, the authors also provide several mitigation methods for these IoAFs. According to these indicators (IoAFs) as well as corresponded mitigation methods, the authors show the effectiveness of indicators (IoAFs) applied on several existing defense approaches and also proposed mitigation methods.
main_review: Pros: 
1. This work propose quantitative indicators to point out reasons of attack failures.
2. The experiments show the effectiveness of proposed indicators of attack failures (IoAFs) and mitigation strategies.

Cons: 
1. Based on [1], we know that IoAFs can work for the correctness of robustness evaluations. However, it only designs metrics for known factors in this paper. It's not surprising that IoAFs are effective. It would be better if the authors have some non-heuristic or theory-driven reasons to explain why this works.
2. The proposed mitigation strategies have already been used in several papers. For instances, $M_2$, $M_4$ and $M_5$ were used in [2, 3]. Therefore, it exists several works showing that the proposed mitigation strategies are helpful.
3. The defense methods included in the experiments were defeated in existing works [4,5].

Overall, quantifying the indicators of attack failure is a novel idea. But the concepts of these indicators have already been proposed before and similar scenario also happened the proposal of mitigation strategies. Besides, the defense methods in the experiments were all defeated in recent works. Namely, these posited mitigation strategies might already exist or at least are pretty common.

[1] Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.

[2] Diversity Can Be Transferred: Output Diversification for White- and Black-box Attacks.

[3] Enhancing Gradient-based Attacks with Symbolic Intervals.

[4] On Adaptive Attacks to Adversarial Example Defenses.

[5] Towards Evaluating the Robustness of Neural Networks.



limitations_and_societal_impact: The authors listed their limitation and meanwhile proposed the alternative method to address it.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a systematic method for understanding the failures of adversarial attacks as means towards thorough evaluation of robustness defenses in the research community. The devised framework relies of three connected aspects: (i) analysis of the main causes why gradient-based adversarial attacks fail, (ii) metrics for identifying their reason of failure and (iii) practical mitigations to ensure attack efficiency when evaluating defenses. All these aspects are illustrated and confirmed on four existing defenses that were initially subjected to limited evaluation.
main_review: Clarity
- Very clear and well-organized paper.
- The code for reproducing the experiments was provided.

Originality:
- The paper formalizes what is common knowledge in the community. However, I do appreciate the effort that goes into this formalism and the necessity for the community to have such a methodology to rely upon.

Quality and soundness:
- The motivation of the paper is clear, and I agree that, despite existing guidelines for robustness evaluation, robustness benchmarks in published research are often incomplete or misleading.
- This work is in line with previous guidelines on robustness evaluation.
- Previous work in the fields of robustness evaluation, broken defenses and machine learning security evaluation platforms are cited appropriately.
- The proposed analysis of four existing defenses is thorough and consistent with the proposed evaluation methodology.

Soundness and significance:
- While I appreciate the proposed framework and its application to the four chosen defenses, a more extensive experimental study would probably be more convincing, as the devised framework is mostly based on observations (as opposed to a formal proof of any kind). There is an abundance of published defenses with incomplete evaluation. Showing the applicability of the contribution to a wider range of defenses would give it better odds of reaching its target: becoming part of the de facto robustness evaluation standard. This suggestion is in line with previous publications proving the lack of robustness in prior methods (e.g., [1, 10] in the paper), where around ten defenses are examined in each.

Minor:
- Fig. 1b: colours seems to be mixed-up between the successful (shown in grey, but should be black) and failed attacks.
- Tab. 1: the explanation for the column name RA should be introduced in the table caption or the paragraph commenting the table.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper reasons about complications that may cause authors to not recognized that their newly proposed defense is ineffective. The authors identify failure motivations, propose 5 indicators (measurable) that are indicative of an attack failure, and also propose appropriate mitigations. The paper very methodically and experimentally presents this findings, and offers a more concrete support towards adaptive attacks evaluations whenever new defenses are proposed. 
main_review: # Strengths
- Very important line of research, focused on supporting and understanding how to systematically carry out adaptive attacks.
- Contributions, significance and novelty is clear.
- Systematic evaluation of indicators of attack failures in existing defense robustness evaluations
- Code scheduled for release. 

# Weaknesses
- Mitigation $M_5$ seems a bit vague in terms of actionable points. 
- The code  link seems broken. Not sure if it was just meant as a placeholder for future publication of code?
- "Robust accuracy" has not been formally defined
- Unclear interpretation of a 'good' or 'bad' value for the indicators fo attack failures (e.g., Figure 1)

# Detailed Comments

The indicators presented are derived from very nice intuitions and quantify effects.  I think this is a very systematic and well-thought work that can improve the state of the art in terms of best practices when evaluating defenses. I do have some comments that could improve the quality of the work. 

## Mitigations

$M_1$ through $M_4$ are quite clear. $M_5$ starts to be a bit fuzzy, but the last paragraph of Section 3 should require more discussion. If I understand correctly, you are suggesting that the whole framework you propose for indicators of attack failure is just for the following:

> the only thing known is that baseline attacks, properly tested, are not working against the defense

What can be the utility of the indicators of attack failures if you try to propose a more advanced adaptive attack upon further investigation?

## Experimental evaluation

The authors conduct a thorough experimental evaluation considering also many existing defenses. However, there are some things that should be better specified:
- **Definitions**. There is no formal definition of "Robust accuracy". 
- **Parameters**. You perform 5 random restarts of the attacks. Is this a number you recommend also future people to use, or is it a good number as long as there are no indicators of attack failures that activate?
- **Interpretation of the indicators**. In Figure 5, you show the average score of the indicators on the X-axis. What I see is for example that the average scores for k-WTA are higher both before and after mitigations than most of the other scenarios (e.g., TWS and Ensemble Diversity). Moreover, it seems you have an average still >0.2 for k-WTA. How should readers interpret such results, and know whether the average score of the indicators is sufficiently low?

## General comment

I may be playing the devil's advocate here, but it would be interesting to have the following discussion:
- Do you feel that an attacker may take advantage of the fact that you proposed the indicators of attacks failure?
In a sense, can an attacker take advantage of the fact that a defender is using indicators of attack failure to evaluate the robustness of their defense? Maybe an attacker that proposes an apparently robust algorithm but of which he knows some other weaknesses?

## Minor comments

- Figure (1) may also use a different line-style because black and gray look a bit similar to my eyes. Maybe also a different line width? Or a greater color difference?
limitations_and_societal_impact: Mostly comments regarding the interpretation of the average scores, and how the indicators may be useful for further adaptive attacks.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper, similar to [1], proposes a systematic way to debug problems that may arise when implementing adversarial attacks to evaluate a proposed defense. They categorize the implementation problems into 4 categories, propose 5 indicator metrics that can identify what problem exists in an implementation, and highlight 5 mitigation strategies that can be used when a particular indicator is triggered. In the context of 4 existing defenses, they show that original Projected Gradient Descent attack implementation overestimates the security of these defenses. By leveraging the proposed indicator values and using their suggested mitigations to properly generate PGD adversarial examples, they are able to show that the attack is much more lethal and can invalidate the existing defenses. They compare the efficacy of their manual approach for debugging with AutoPGD, which tunes hyperparameters automatically to improve the PGD attacks chances of success.

[1] Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., ... & Kurakin, A. (2019). On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705.
main_review: ### Strengths

- The paper is easy to follow.
  1. Provides an abstract view of adversarial attacks and shows where implementation downfalls arise,
  2. Proposes a set of indicators to identify these downfalls,
  3. Suggests which existing mitigation strategies to leverage when a indicator triggers,
  4. Shows the efficacy of this manual process to debug attack implemenation by comparing it to automatic approaches.

- The paper treats a subset of the issues highlighted in [1] in a quantitative fashion by proposing indicator functions.


### Weaknesses / Correctness / Clarification

- The main novelty of the paper seems to be proposing the indicator functions. Note that the universal skeleton of attack algorithms and the mitigation strategies are already known. Perhaps, tying them together in the context of the indicator functions (i.e. parts of the algorithm that trigger it and mitigations that need to be used) is another novelty.

- On the indicator functions, I have a few concerns, clarifications and perspectives. As an overall comment, I suggest you spend more time describing the challenges you faced to come up with these metrics, describe them with more rigor and highlight how it can take care of edge scenarios (Many of the figures/tables have redundant information; hence, space should not be a problem). More specifically,
  - I_1 (silent success) and I_5 (non-transferability) are zeroth-order strategies that are often used by researchers who try to debug the efficacy of an attack algorithm. While these simple measures do show that not all researchers did due diligence when evaluating defenses, I am not convinced that these indicators address any challenge-- it is good software development guidance.
  - For I_2 (break-point angle), the authors state "we conclude the shape by considering the point of the loss curve that is further to such conjunction" (lines 168-169). I unfortunately did not understand this. I wanted to evaluate my understanding on some cooked up complicated loss plots but couldn't as I wasn't sure how to concretely identify this third vertex from the statement above.
  - For I_3 (increasing loss), is there some sort of formal way to say that this measure of area will always be (an admissible) heuristic for evaluating if the attack can converge to a good local optima? While for figure 4 it does make sense, I am not sure if it is always going to work. Even a proof for a weaker statement like _Given two attacker loss plots (for the same model, data blah..), if one converges to a better local optima than the other, the value of I_3 will always be smaller for it_ should work.
  - For I_4 (zero gradients), again some sort of formal treatment + highlight what was the major challenge one needed to solve to come up with it would be helpful.

- From table 1, I see that APGD has higher values for several indicator functions, such as Distillation I_2 (0.4), Ens. div. I_2, TWS I_2 then the mitigated PGD*. Yet, in all 3 cases, Auto PDG is almost similar or better than PGD*. This makes me question the validity of I_2 as a good indicator. Similarly for I_3, APGD has almost the same value as Ensemble Div for APGD (0.14) and PGD* (0.15) and yet APGD is much more effective at mitigating implementation issues. As stated above, without some formal guarantees, it is hard to accept them as indicators on face value.

- Evaluation against AutoPGD shows that this manual effort is better only in the context of debugging attack implementation for k-WTA. For others, its almost it's either a bit worse (ensemble div) or similar (distillation, TWS). At this point, I am not convinced why should one rely on manual debugging techniques (which may have pitfalls of their own).

- As all gradient based attacks (eg. CW, PGD, BB, DDN) can be represented as part of their framework, wondering why evaluation only looks at PGD? They should create an auto-version of the other attacks and adapt them as baselines to compare against. Simply showing efficacy of indicators based on only one attack type (i.e. PGD) isn't enough.

- The indicators provide a user confidence that many obvious issues do not exist but is not sufficient to say that having mitigated all the issues with the attack implementation, if the defense is really resilient (217-219).

- Table 1 shows a subset of indicators are used for each defense-attack combo, but Table 2 (sort of) implies all mitigations are applied in a step-by-step fashion. Not sure if a particular sub-set of indicators are triggered, why are all mitigations applied? If not, please ensure the tables don't mislead the reader.

- The (pretty) figures (Fig 5 and Fig 6) seem redundant. All of this info is already there in the tables (or am I missing some critical insight here?).

#### Minor

- (245) Streng`ht` -> Streng`th`
- (Table 1) Replace `,` with `.` in accuracy values 58`,`2% -> 58`.`2%
- (249) defen`ded` -> defen`ces`
- (301) 6`,`4% -> 6`.`4%


[1] Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., ... & Kurakin, A. (2019). On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705.
limitations_and_societal_impact: See main review.
review_title: null

Meta-review: All reviewers acknowledged the usefulness of the proposed indicators for debugging attack failures, but at the same time, there are major concerns being raised regarding overlapping contributions to known literature and insufficient experiments for demonstrating the value of the indicators. The authors' rebuttal did not address the main concerns of the majority of the reviewers.

Here is a summary of the main concerns. I hope the authors find the review comments and summary useful for preparing the future version.

1. The introduced indicators are useful in debugging attack algorithms, but many of them are common practices or overlap with known results in existing literature, limiting the contributions of this work
2. Some indicators overlap with existing literature on adaptive attacks, such as the NeurIPS 2020 paper "On Adaptive Attacks to Adversarial Example Defenses"
3. This paper mainly focused on 4 defenses (that were previously shown to be broken) while earlier works of similar fashion such as "Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples" and "On Adaptive Attacks to Adversarial Example Defenses" show defense failures on about 10 defenses.
Decision: Reject