Evaluating Model Robustness Against Unforeseen Adversarial Attacks

Authors: Maximilian Kaufmann, Daniel Kang, Yi Sun, Xuwang Yin, Steven Basart, Mantas Mazeika, Adam Dziedzic, Franziska Boenisch, Tom B Brown, Abhinav Kommula, Oliver Zhang, Akul Arora, Jacob Steinhardt, Dan Hendrycks
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: AI Safety, ML safety, adversarial robustness, distribution shift, unforeseen adversaries

Abstract:
When considering real-world adversarial settings, defenders are unlikely to have access to the full range of deployment-time adversaries, and adversaries are likely to use realistic adversarial distortions that will not be limited to small $L_p$-constrained perturbations. To narrow in on this discrepancy between research and reality we introduce ImageNet-UA, a new benchmark for evaluating model robustness against a wide range of unforeseen adversaries. We make use of our benchmark to identify holes in current popular adversarial defense techniques, highlighting a rich space of techniques which can improve unforeseen robustness. We hope the greater variety and realism of ImageNet-UA will make it a useful tool for those working on real-world worst-case robustness, enabling development of more robust defenses which can generalize beyond attacks seen during training.

Decision: reject
Reviewer scores: 5, 3, 8, 3

Reviews and discussion:
<thread>
creation date:
1729693423965

thread_type:
official_review

<thread_object>
creator: reviewer_8hc1

<content>
summary:
This paper propose a benchmark of adversarial attacks computed on different noise patterns.
These are made differentiable, and applied to images with PGD. Models are adversarially trained under a specific perturbation, and evaluated with others to test whether they have generalized across unseen attacks.

strengths:
**Models should be robust to whatever noise it appears.** I agree with the authors that model must be resilient to *any* perturbation of images, and testing more attacks is always useful.

**Robustness augments when diversifying the noise.** It is interesting to notice that models behave better when their training is enriched with the various noise patterns. However, this is somehow trivial due to the formulation of adversarial training: the better the approximation of the inner maximization is, the more robust the model it gets. Still, the authors did the right experiments, showing that this is empirically proved also in this setting.

weaknesses:
**The presented strategies are L$_p$-bounded attacks.** I disagree with the authors stating that the proposed attacks are beyond the L$_p$ norm. In particular, it can be still computed the amount of perturbation and bounded to the maximum budget. The whole optimization problem can also be written as an L$_p$ problem, and constrained to be such a problem. Maybe the projection is a bit different, because the authors want to use specific patterns. Hence, all the discussion regarding going beyond that threat model does not apply on this paper.

**No details on the attack.** The authors spend a lot of space to present the manipulations, and they vaguely state they are differentiable. However, I could not understand how. This content *must* be present in the main paper, since it is a claimed contribution. Placing them in the appendix is not the right thing to do. Hence, I strongly suggest the authors to remove figures (like 2 and 4, they are not useful to understand the efficacy of the methods) and expand on the technical details of the attack.

**Debatable contribution of manipulations.** I debate that the nine attacks are novel, since they are all well-known noise patterns already present in the literature. Also, JPEG compression has been extensively used a lot (also on the defensive side). I would frame that the real contribution is how the authors attach those noise patterns in a differentiable way (see previous comment). Hence, 3.3 is not really novel per se to the extent of my knowledge.

**Unfair analysis.** The authors show that models are not robust to new manipulations. However, the authors should compare the efficacy of attacks at the same perturbation budget (same norm) and same number of queries to the models (i.e. forward and backward). Otherwise, in absence of those, attacks can be easily biased towards being effective or not. Also, why the authors did not use AutoAttack, similarly to RobustBench? Also, the authors do not provide their implementation of PGD, which might also contain bugs.

**Confusing presentation.** There is a digression on L$_2$ adversarial training which is not clear to me why is discussed. Also, L$_2$ is still an L$_p$ perturbation, and the authors might want to further investigate this by studying *why* they obtained their results. Still, I did not understand the connection in comparing different threat models in that portion of the paper.

questions:
1. Can the authors specify the amount of perturbation injected, so that it is comparable with PGD?
2. Can the authors test AutoAttack, with the same level of noise used for their perturbations?
3. Can the authors better clarify the novelty of their approach, by revising the content and clearly specify how they produce differentiable attacks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729795124169

thread_type:
official_review

<thread_object>
creator: reviewer_espk

<content>
summary:
This paper introduces a set of 19 new non-Lp attacks for measuring unforeseen robustness.  The authors introduce a score called UA2 which measures robustness across a set of 8 of these attacks for benchmarking existing models.  Additionally, they provide experiments to demonstrate that robustness against these non-Lp attacks is directly correlated with Lp attack robustness or distribution shift robustness.  The authors also evaluate several defenses including combining adversarial training with image augmentation (PixMix), multiattack training, and perceptual adversarial training.

strengths:
- The authors propose many new attack types which can be used for measuring unforeseen robustness
- Extensive experiments measuring the performance of the current state of research against the set of attacks proposed
- Writing is clear

weaknesses:
Overall, I thought this was a strong paper.  One thing that could be incorporated for strengthening the paper could be adding some evaluations on the attacks left out of the 8 core attacks and demonstrating that the performance measured over the left out attack set is correlated with performance measured over the 8 core attack set.  This would empirically demonstrate that the performance on the 8 core attacks are highly representative of the 19 attacks.
 
Minor typos:
- Table 1 caption: "disctinct"
- line 318: $\varepsilon_a$ notation as opposed to $\epsilon_A$ used elsewhere

questions:
- In section 4.1 comparison to ImageNet-C what does "non-optimized" version of the attack mean?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730130519310

thread_type:
official_review

<thread_object>
creator: reviewer_1onw

<content>
summary:
The authors propose ImageNet-UA, a benchmark for evaluating model robustness against unforeseen adversarial examples, i.e., adversarial perturbations not considered in the design phase of the model. Supposing the models' developers do not have access to worst-case adversarial examples during training, the benchmark proposes a set of non-Lp gradient-based attack methods as worst-case inputs to evaluate the model's robustness at test time. The authors provide an in-depth analysis of various adversarially trained models on different datasets from the benchmark.

strengths:
- The authors designed and evaluated 19 non-Lp adversarial generation methods. 
- The authors proposed a wide analysis of the benchmark on several models and different datasets.

weaknesses:
- The authors did not compare the benchmark to other $L_p$ attack methods in the state of the art. Additionally, there is significant work that should be included in the comparison or at least mentioned in the related work
- The author’s evaluation of their adversarial training method is incomplete.  
- The authors introduced a new metric UA2 (Unforeseen Adversarial Accuracy), but it is unclear why the new metric is necessary. 

## Comments

1. The authors did not compare the benchmark to other $L_p$ attack methods in the state of the art. Additionally, there is significant work that should be included in the comparison or at least mentioned in the related work

In section 4.1 the authors compare UA2 to other distribution-shift benchmarks, and they claim, based on the results (Table 5), that UA2 is a measure of worst-case robustness, like $L_p$, but they did not assess the distributional shift of $L_p$ perturbed samples. 
Additionally, there have been recent workshops and efforts to address the real-world issues applied to ImageNet, and the authors should provide some references and possibly differentiate their work from these. One example is the ShiftHappens workshop at ICML 2022 https://shift-happens-benchmark.github.io

2. The author’s evaluation of their adversarial training method is incomplete.  

In the Experiments section, there is a broad discussion on how UA2 robustness, based on the evaluation of unforeseen adversarial example, is distinct from  $L_p$ robustness. However, the authors based this claim on the evaluation of just one $L_p$ attack method (PGD) and just on $L_\infty$  bound constraints, with a fixed number of optimization steps (only $50$, very limited for ImageNet, and without providing evidence of convergence).  

3. The authors introduced a new metric UA2 (Unforeseen Adversarial Accuracy), but it is unclear why the new metric is necessary. 

The authors introduce the UA2 (Unforeseen Adversarial Accuracy) metric as the average Robust Accuracy achieved by the model over the non-$L_p$ methods in the benchmark. Although this metric provides an overall score relative to the benchmark, it doesn't allow for a thorough assessment of which attacks are truly effective (and thus worst case) and which are not. For example, the results with 'medium distortion,' shown in Figure 8, clearly indicate that three types of attacks (Elastic, Wood, Gabor) generally perform better than the others. 
Additionally, it's not clear why the authors use the average in this metric rather the worst case (so the perturbation that has the strong effect) Unforseen Adversarial, which would be an ensemble attack composed by all the attacks rather than single evaluations.

#### Additional comments:

In sect. 3.3, the authors select eight attacks out of the nineteen, but they don't give a justification on why they selected these. Specifically, is there a possibility that other, non-selected attacks, are super effective against specific models? A more substantial contribution would be to understand in which cases (e.g., for which preprocessing techniques or for which type of models) one attack works better than the other. As an example, a useful actionable insight would be "one can exclude attack A, B, C as the preprocessing technique would remove such perturbation". This insight gathering is lacking in the paper but it would be much more useful to enhance robustness evaluations to unforseen attacks.

In sect. 3.3, the authors suggest quantization only in the JPEG compression strategy, however it's still to be observed that, given that the benchmark is specific to the image domain, in theory all the manipulations should still produce images (thus quantized in integer values in [0, 255]). This aspect seems overlooked in the paper (at least in the text).

In section 4.2 the authors propose a combined adversarial training approach that consider $L_\infty$ adversarial training and "PixMix" augmentation. The authors' analysis of PixMix is incomplete in two ways. First, they don't explain what PixMix is or provide any reference. Second, in Table 6, they only show improvements in the UA2 metric, without evaluating how this adversarial training method affects the $L_p$ robustness. 

Generally, the authors evaluate the robustness of models adversarially trained on a specific kind of $L_p$ perturbation, against attacks that optimize for a different type of $L_p$ perturbation. Indeed, the benchmark is composed of non-$L_p$ methods which optimize the adversarial sample considering both $L_\infty$ and $L_2$ constraints. When comparing with $L_p$ attack methods the authors considered only $L_\infty$.  


#### Minor issues:

- there is a mixed use of american and british english (e.g., defenses and defences). Please, double check and remain consistent throughout the manuscript
- Figure 3 could benefit from a more structured presentation. Specifically, naming the steps and providing a step-by-step overview would help understand the approach better

questions:
1. Could you argue why certain non-$L_p$ attack methods are more effective than others? 

2. Could you report the $L_p$ robustness of the models adversarially trained with $L_\infty$ and with $L_\infty$ + PixMix? 

3. Could you report the results for an increased number of steps? Or please, provide evidence of convergence for the attacks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730550340893

thread_type:
official_review

<thread_object>
creator: reviewer_afdd

<content>
summary:
This paper introduces the ImageNet-UA benchmark and the UA2 metric to assess model robustness against a variety of unforeseen adversarial attacks. Experimental results indicate that traditional Lp robustness training may have limitations when addressing unknown attacks. The paper suggests employing L2 adversarial training combined with image augmentations to enhance model performance under these unexpected attack scenarios.

strengths:
1. The paper conducts comprehensive experiments using a set of 19 diverse non-Lp attacks, offering a thorough evaluation of model robustness in the face of various unforeseen adversarial scenarios. 

2. By introducing the ImageNet-UA benchmark, the paper fills a gap in adversarial robustness research, enabling a more applicable assessment of model resilience to real-world adversarial attacks beyond traditional Lp constraints.

weaknesses:
1. A considerable portion of the logical content is relegated to the appendix, which is suboptimal. Key elements of the logic should be presented within the main text. For instance, Figure 5 should either be compressed and included in the main paper or replaced with an equivalent narrative explanation.

2. In Figure 3, the term $m \times m \times 2$ lacks a corresponding definition, which should be clarified.

3. Equation 2 suggests that the adversarial attack should aim to maximize the loss function with respect to perturbations, rather than as currently presented.

4. The study mainly employs various image augmentation techniques to examine model performance without contributing significant advancements in model verification methods (i.e., assessing stability across augmented samples), which falls short of the standards for ICLR.

questions:
Refer to Weaknesses
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces ImageNet-UA, a benchmark designed to evaluate model robustness against unforeseen adversarial attacks that extend beyond traditional Lp-bounded threat models. It introduces UA2 to measure a model robustness across these diverse attacks. The benchmark includes 19 non-Lp attacks, from which 8 are highlighted as core attacks. The study demonstrates that combining adversarial training (particularly L2 adversarial training) with image augmentation techniques can enhance model robustness under unforeseen attacks. 

The paper makes a contribution in building a benchmark to evaluate robustness under unseen attacks, which is a longstanding issue in adversarial robustness research. The paper conducts extensive experiments with 19 attacks, providing insights into robustness under unforeseen scenarios. However, the paper has several weaknesses.

- Novelty: The attacks used in the benchmark are not entirely novel, as they are based on well-known noise patterns from existing literature.
- Metric: The necessity and effectiveness of the UA2 metric are unclear. Questions arise about why it averages performance rather than focusing on the worst-case scenarios.
- The selection of 8 core attacks out of 19 is not well justified.
- The evaluation is limited to adversarial training models. Could you consider other generalizable defenses, e.g., diffusion-based defenses.

The paper received mixed reviews with one accept recommendation, one borderline reject recommendation, and two reject recommendations. After author-reviewer discussion and AC-reviewer discussion, the paper still has some weaknesses as listed above. Therefore, the AC considers that the paper falls short of the ICLR acceptance threshold and recommends rejection.