Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a method to test if an adversarial defense method for deep neural networks is strong enough. By a simple modification of the threat model, the paper introduces a new classifier in which the existence of an adversarial example is guaranteed. Such a modified model can be used to examine whether an attack is strong enough, and hence the evaluation of robustness is convincing. Experimental results show most of the previously published defenses are insufficiently strong.
strengths_and_weaknesses: Originality & Significance:

The idea of the paper is very interesting and looks novel to me. The results show that most of the previously published defenses are not strong enough under the proposed test, which is a little bit surprising. The paper is therefore of significance in that it could guide future research on this area to consider stronger evaluations.

Quality & Clarity:

The paper is generally well-written. The analogy to refuting proof in complexity theory is interesting. The legend of Figure 1B is not very clear.
questions: In L139:
 >If this is not possible for an original sample xc, we cannot apply the test and, hence, skip the sample

What is the proportion of samples that are skipped in the experiment?
limitations: The author has discussed the limitation of the paper in Section 5.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies adversarial robustness evaluation of defense models. Based on the fact that the accurate robustness evaluation is difficult, this paper proposes a new binarized testing method that can discover weak attacks. Eleven out of thirteen defenses fail the test and could be broken by stronger defenses.
strengths_and_weaknesses: This paper studies an important problem and make a great contribution to the field. Although previous studies have been conducted to provide guidelines and practice to develop more accurate robustness evaluation, there truly lacks a formal test whether the robustness is overestimated. This paper proposes a novel testing method, which successfully identifies weak defenses.

The main weakness of this paper is that most adopted defenses have been broken before. Thus it is not surprising to see these defenses cannot pass the test. Evaluating more recent and state-of-the-art defenses can make the paper more convincing. The generalizability of the proposed method for more models is questionable. 
questions: The proposed binarized test also needs to be designed for different models. How about the efforts?
limitations: The authors have discussed the limitations and potential negative societal impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Hundreds of adversarial attacks and defenses have been proposed in the last few years. How Hundreds of adversarial attacks and defenses have been proposed in the last few years. How robust defense depends on the choice of evaluation. All defenses can be broken given enough perturbation. This paper presents a model, defense, and attack agnostic methodology to identify weak defenses. The authors propose a binarization test. (1) Create a 2-class synthetic dataset based on real examples which are in-boundary and on the boundary
 (2) Take the feature extractor of a robust classifier (f*) and train a binary classifier (g) that can classify the dataset (3) Evaluate the attack on the h = (g o f*) classifier and compute the % of times the attack is successful. Higher this score, more potent the attack. 11/13 previously published defenses failed this test, showing that the defenses are not strong but the evaluation is weak.

strengths_and_weaknesses: Strengths: The authors address an important issue of robustness evaluation in the adversarial literature. The binarization test technique is novel and it is both simple and computationally cheap. The authors discuss the failure case, I appreciate it.

Weakness: The writing is very unclear. The idea is simple, but it is really hard to understand from the writing. Things like figures and even algorithms that are supposed to convey the message easily are hard to parse. It needs a major re-write to simplify things if it were to be accepted into the conference. 
questions: - How many samples are you using in training the classifier g? Do you have any ablation on this? 
- Do you have any ablation on the strength of the attack $\epsilon$ and how the attacks fare against the binarization test?
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a binarization test to identify weak attacks against adversarial example defenses. The proposed test changes the model’s prediction layer to a binary classifier and fine-tunes it on a small crafted dataset for each benign example. As a result, the original attack, if sufficiently strong, should be able to find adversarial examples when applied to the modified model. This test serves as an active robustness test to complement existing passive tests of weak attacks. Empirical results show that the proposed test effectively re-confirmes the weak evaluations of 11/13 previous defenses, and two of them were not discovered before.
strengths_and_weaknesses: ### Originality

**Strengths (major)**
* The proposed active test provides a novel perspective for identifying weak evaluations of adversarial examples defenses.
* The overall idea and approach are novel and insightful.
* The design of evaluating the difficulty of the proposed test is good, as it provides validation when a weak attack passes the test.

**Weaknesses (minor)**
* **Unclear algorithmic improvements to [34].** The authors mentioned in footnote 1 (page 3) that a similar idea was used before as a honeypot defense [34]. It is suggested to discuss how the algorithm (e.g., injecting adversarial examples) used in this paper differs from those used in [34]. For example, are there any new challenges when directly adopting the previous approach, are there any insightful modifications to that approach so it fits the proposed test, or does the idea of [34] fit more in detecting weak attacks rather than as a defense?
* **Lower bound for Equation (1).** Does Equation (1) need a lower bound for the inner point sampling to guarantee the minimum hardness of the test?

### Quality

**Strengths (major)**
* The evaluation is strong and clearly demonstrates the effectiveness of the proposed active test.
* Two previous weak evaluations are discovered.

**Weaknesses (minor)**
* **Unclear effect of changing the classification head.** While it makes sense as a test to change the prediction head, I am not sure if that would negatively affect the test as the changed model may not precisely “mimic” the original model. For example, if we were to evaluate adversarial training, the fine-tuned prediction head may lead to a less robust model than the original one. As a result, it seems easier to attack the test model than the original one.
* **Applicability to adversarial training.** While I understand that evaluating skeptical defenses is more straightforward in demonstrating the effectiveness of the proposed test, it is suggested to discuss the proposed test’s applicability to adversarial training. In particular, I am curious if the test could detect weak attacks (e.g., weak PGD with a large step size or a few steps) on adversarially trained models (with different robustness). This might be informative as the robustness from adversarial training is also developing, and weak attacks (PGD vs. AutoPGD) may overestimate its robustness.
* **Unclear overheads.** It is suggested to include the overheads of Algorithm 1, as it trains a new prediction head for each test sample. It is also unclear how many test samples are needed to produce a confident claim of passing the test. At L138, the authors mentioned that some samples might not produce a model with perfect accuracy. I am curious why this would happen and if that affects the overheads significantly.
* **Slightly confusing clarification of false positives.** The clarification at L261-263 is slightly confusing, as I did not expect misclassified samples to be included in the evaluation from the beginning. If included, it may not be the attack that "is not used correctly," but the evaluation (of this paper) should follow the same setting as the integrated attack. This further leads to the confusion of why the label matters here: If all points are sampled without labels in Equation 1, and the RunAttack in algorithm 1 refers to the exact original attack with the exact choice of label, then it seems that we should not observe the artifact at L265-268.

### Clarity

**Strengths (major)**
* The paper is generally well-written.

**Weaknesses (misc)**
* The legend for the threat model in Figure 1 is confusing; maybe it should be an empty square.
* I feel that some contributions can be highlighted earlier in the paper. For example, the inclusion of detection defenses and the newly identified flaws.
* Most functions in Algorithm 1 are not clearly defined and may need some clarification or reference to corresponding texts. For example, RunAttack, RunRandomAttack, SampleInnerPoint, TrainReadout, etc.
* The statement at L190-192 is hard to follow and needs more motivation and clarification.

### Significance

**This paper provides strong positive results for more confident evaluations of adversarial example defenses. While I have listed several weaknesses above, most are not major and are suggested as discussions that further strengthen the paper.**
questions: I would recommend that the authors elaborate on the four weaknesses in the Quality section.
limitations: The lack of evaluation of adversarial training might be a potential weakness, but the existing evaluation is already sufficient and representative.
review_title: null

Meta-review: metareview: This paper proposes a simple yet effective test to identify weak adversarial attacks, and thus weak defense evaluations. Empirical results have revealed insufficiently strong evaluations in 11/13 previous published defenses. To me, the paper studies an important problem and makes a valuable contribution to the active research field of adversatial defense and robustness evalution. I recommend acceptance, and encourage the authors to incorporate the reviewers' comments and suggestions when working on the final version.
confidence: Certain
Decision: Accept