Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary

The authors propose ImageNet-UA, a framework to evaluate the
effectiveness of test-time unforeseen adversarial ML attacks. In this
particular context, the authors observe that attackers are not limited
to induce minimal Lp-norm perturbations and therefore it is important to
understand how robust existing defenses are against attacks that diverge
from these constraints while still being realizable. Within their
framework, the authors propose four novel adversarial attacks to
consolidate our understanding of test-time adversarial ML attacks.

## Strengths

+  Interesting intuition that test-time adversarial attacks are not all
necessarily driven by minimal perturbations
+  Attack-evaluation framework available to the community

## Weaknesses

-  Recent work explores the need for having a unified theoretical
   framework to reason about realizable attacks
-  Actual benefit of the approach

## Comments

This is an interesting paper that explores an avenue that is often
neglected in the context of test-time adversarial ML attacks. There is a
tendency on focusing on attacks that minimize the perturbation in a
given Lp-norm, when in fact there are several application domains for
which this constraint does not necessarily matter. Here, in the context
of realizable or problem-space attacks, one is required to reformulate
the problem, which may include other constraints attacks must satisfy.

As the authors admit, ImageNet-UA is not exhaustive although it tries to
evaluate the robustness of defenses over diverse unforeseen test
(adversarial) distributions. Under this premise, I somehow fail to see
the real benefit of such a framework, unless the authors show such
additional attacks comprise widely-used transformations that should
therefore become part of the threat model of any adversarial work in the
problem space. (In addition, most of the attacks seem to be a marginal
improvement of existing ones.) Conversely, a theoretical reformulation of the
problem-space would have been perhaps more useful. That would have
allowed one to reason about properties or set of transformations one
could or should consider, along with other constraints, as outlined in
[1]. 

I also wonder what's the rationale behind the mUAR metric. It is
definitely useful to have a way to capture the robustness of models
against unforeseen attacks, but I wonder whether an average as opposed
to, e.g., a geometric mean, would provide useful insights. In a way, the
average would consider all the attacks equally challenging where, in
reality, this might be the case. (I do actually like the use of mUAR and UAR tho,
as outlined in Section 5.2).

## Additional comments

In Section 4, the authors explain how the distortion size \epsilon_max
is chosen: "[...] the smallest ε which either reduces adversarial
accuracy of an adversarially trained model at distortion size ε below 25
or yields images confusing humans". How is this verified? Do the authors
rely on a user study? Does this only rely on Gilmer et al. (2018)?

[1] https://s2lab.kcl.ac.uk/projects/intriguing/ (IEEE S&P 2020)
review_title: ICLR 2021 Conference Paper2332 AnonReviewer4 Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Update**: Thanks to the authors for addressing my comments and releasing the source code. Code is well-structured and easy to follow. It can be definitely used as a supplement for the robustness evaluation. However, judging by the implementation, the rain, snow, and fog attacks are too simplistic. From the paper, it is not clear how well these attacks approximate the respective type of perturbations in real-world scenarios. Therefore, the proposed set of 6 attacks is more the author's heuristic than something that can be used for the evaluation of computer vision systems, e.g. autonomous vehicles. Given all that, I decrease my score from 5 to 4.

###### Summary
The paper proposes a novel benchmark for the evaluation of the model's robustness against unforeseen adversaries. The authors suggested using 6 types of adversaries: JPEG, FOG, SNOW, and GABOR attack, L1, ELASTIC. Fully-differentiable variants of the above attacks are based on the previous work. The authors introduced the novel $l_1$-norm attack, which uses the Frank-Wolf algorithm to satisfy $l_1$-perturbation constraint. The authors introduced two metrics to compare defenses: robustness against a single unforeseen attack and mean unforeseen robustness. Experiments with adversarially trained models and non-adversarially trained models align with the previous work in the area.

###### Reasons for score: 
I vote for a weak accept. The paper studies an important problem of the robustness of unforeseen adversaries. Introducing novel benchmarks beyond $l_{p}$-norm robustness is an important problem. The paper is clearly written and easy to follow. The evaluation results are extensive. However, implementation details of some attacks (SNOW, FOG, GABOR) are missing. JPEG attack is based on the previous work. It is not clear if the authors plan to release the benchmark suite to the reviewers. The most similar work in Hendrycks et al. 2019 uses a more diverse set of attacks.

###### Concerns:
- Novel differentiable attacks are based on previous work, e.g. differentiable JPEG and FOG. The algorithm and implementation details for SNOW and GABOR attacks are not provided. It is not clear how to reimplement these attacks.
- The paper is the most similar to Hendrycks & Dietterich 2019. The main difference is the use of differentiable attacks. However, previous work in Hendrycks uses a larger set of corruptions. This work includes only 5 corruptions from Hendrycks et al. 2019. Do the authors plan to include other types of attacks for the Imagenet-UA benchmark?
- It is not clear from the main text if the authors plan to release the benchmark source code during the reviewing process.
  
###### Minor comments:
- Comment that Elastic $l_1$ attack is based on heuristics might be incorrect. Elastic $l_1$ attack uses a proximity operator, a principal way to minimize $l_1$-norm.
review_title: The paper proposes a novel benchmark for evaluation of the model's robustness against unforeseen adversaries. The authors should provide implementation details of the novel attacks and make the source code available during the reviewing process.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes four novel efficient adversarial attack methods beyond Lp threat models. Together with other two existing attack methods, these six attack methods combine as a framework to evaluate robustness of defenses against unforeseen attacks. In this framework, the novel measure is normalized with the performance of adversarial training. The experiments show that the Linf adversarially trained model may not lead to improvement of robustness against other threat models. It is expected that the framework could help test model robustness.

## Advantages

- Evaluating model robustness against unforeseen adversaries is an important problem.
- The four proposed adversarial attacks are differentiable and easy to use.

## Disadvantages

- The proposed measure mUAR is quite heuristic, e.g., the choice of the epsilon parameter.
- As a baseline framework, it is hard to say that the six adversarial attacks are sufficient to evaluate robustness against unforeseen adversaries. As a result, the claims seem not to be very convincing.
- In general, the result that adversarial training is not robust to unforeseen attacks is not surprising and has be been discussed a lot in literature.  It would be good to evaluate more existing defenses in order to derive more novel, interesting and inspiring insights.


review_title: Good motivation but not unconvincing enough

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents the existence of several adversarial attacks that have meaningful visual concepts. Based on these attacks, the authors further develop a method to measure the robustness of a network or an adversarial defense. 

Despite being conceptually interesting, this paper is working on the thing that many studies have done. The related work section is not on the point, in my opinion, the core problem (''unforeseen adversaries'') of this work should be ''adversarial attacks that are designed to break the commonly employed defenses'', but not ''adversarial attacks that are visually meaningful, and occasionally break the commonly employed defenses''. Therefore, the related work should focus on these studies that propose measures of robustness, and the experimental analysis should focus on doing a comparison of these measures, telling why the readers should adopt the ImageNet-UA and mUAR as their baseline, but not others.

On the other hand, the analysis of these newly developed is not sufficient, and the experimental results can not support the claims convincingly. For instance, in Table 3, the JPEG attack acquires the worst performance and the data augmentation methods do not help in all scenarios. Meanwhile, the Fog and Snow attacks are less effective than the $L_\inf$ and $L_2$ and benefit from more data. This implies they have different dynamics thus should not be used for mUAR homogeneously.

Anyway, I suggest the authors carefully revise the motivation and experimental analysis.
review_title: An interesting idea

Meta-review: In this paper, a defense method against test-time adversarial ML attacks is proposed. Unfortunately, it is not clear whether the proposed method is practically useful or not, because the types of attacks assumed in this paper are too simple and heuristic. Also, the position of the proposed method in the vast amount of existing research on adversarial attacks is not clear. Although the proposed method is conceptually interesting, no evidence is provided that the proposed method is significantly superior to existing approaches.
Decision: Reject