Testing Robustness Against Unforeseen Adversaries
Authors: Daniel Kang
Yi Sun
Dan Hendrycks
Tom B Brown
Jacob Steinhardt
Venue:  (2021)
Keywords: adversarial examples
adversarial training
adversarial attacks

Abstract: Most existing adversarial defenses only measure robustness to $L_p$ adversarial attacks. Not only are adversaries unlikely to exclusively create small $L_p$ perturbations, adversaries are unlikely to remain fixed. Adversaries adapt and evolve their attacks; hence adversarial defenses must be robust to a broad range of unforeseen attacks. We address this discrepancy between research and reality by proposing a new evaluation framework called ImageNet-UA. Our framework enables the research community to test ImageNet model robustness against attacks not encountered during training. To create ImageNet-UA's diverse attack suite, we introduce a total of four novel adversarial attacks. We also demonstrate that, in comparison to ImageNet-UA, prevailing $L_\infty$ robustness assessments give a narrow account of adversarial robustness. By evaluating current defenses with ImageNet-UA, we find they provide little robustness to unforeseen attacks. We hope the greater variety and realism of ImageNet-UA enables development of more robust defenses which can generalize beyond attacks seen during training.

Under review as a conference paper at ICLR 2021
TESTING ROBUSTNESS AGAINST UNFORESEEN AD- VERSARIES
Anonymous authors Paper under double-blind review
ABSTRACT
Most existing adversarial defenses only measure robustness to L, adversarial attacks. Not only are adversaries unlikely to exclusively create small Lp perturba- tions, adversaries are unlikely to remain fixed. Adversaries adapt and evolve their attacks; hence adversarial defenses must be robust to a broad range of unforeseen attacks. We address this discrepancy between research and reality by proposing a new evaluation framework called ImageNet-UA. Our framework enables the research community to test ImageNet model robustness against attacks not encountered during training. To create ImageNet-UA's 's diverse attack suite, we introduce total of four novel adversarial attacks. We also demonstrate that, in comparison to ImageNet-UA, prevailing Loe robustness assessments give a narrow account of adversarial robustness. By evaluating current defenses with MnageNet-UA, we find they provide little robustness to unforeseen attacks. We hope the greater variety and realism of ImageNet-UA enables development more robust defenses which can generalize beyond attacks seen during training.
INTRODUCTION
Neural networks perform well on many datasets (He et al. 2016) yet can be consistently fooled by
minor adversarial distortions (Goodfellow et al., 2014). The research community has responded by quantifying and developing adversarial defenses against such attacks (Madry et al.) (017). but these defenses and metrics have two key limitations.
First, the vast majority of existing defenses exclusively defend against and quantify robustness to Tonsconstr attacks (Madry al., 2017 Cohen et al. 2019 Raff et al. 2019: Xie et al.) 2018). Though real-world adversaries are not I. constrained (Gimer et al. 2018) and can attack with diverse distortions (Brown et al. 2017 Sharif et al. 2019). the literature largely ignores this and evaluates against the Lp adversaries already seen during training (Madry et al. 2017 Xie et al. 2018), resulting in optimistic robustness assessments. The attacks outside the Lp threat model that
have been proposed (Song et al.) [018]] Qiu et al. 2019 Engstrom et al. 2017) Evtimov al. 2017 Sharif et al. (016)) are not intended for general defense evaluation and suffer from narrow dataset applicability, difficulty of optimization, or fragility of auxiliary generative models.
Second, existing defenses assume that attacks are known in advance (Goodfellow) 2019) and use knowledge of their explicit form during training (Madry et al. 2017). In practice, adversaries can deploy unforeseen attacks not known to the defense creator. For example, online advertisers use attacks such as perturbed pixels in ads defeat ad blockers trained only on the previous generation of ads in an ever-escalating arms race (Tramer et al.) 2018). However, current evaluation setups implicitly assume that attacks encountered at test-time are the same as those seen train-time, which is unrealistic. The reality that future attacks are unlike those encountered during training akin a train-test distribution mismatch-a problem studied outside of adversarial robustness (Recht et al. 2019 Hendrycks & Dietterich 2019) but now brought to the adversarial setting.
The present work addresses these limitations by proposing an evaluation framework ImageNet-UA to measure robustness against unforeseen attacks. ImageNet-UA assesses a defense which may have been created with knowledge of the commonly used L or L2 attacks with six diverse at- tacks (four of which are novel) distinct from Loo or Lg. We intend these attacks to be used at test-time only and not during training. Performing well on ImageNet- UA thus demonstrates gen- eralization diverse set of distortions not seen during defense creation. While ImageNet-UA

---

Under review as a conference paper at ICLR 2021
Previous Attacks
Lo00
L
L1
Elastic
Our New Attacks
JPEG
Fog
Snow
Gabor
Figure Adversarially distorted chow chow dog images created with old attacks and new attacks. The JPEG, Fog, Snow, and Gabor adversarial attacks are visually distinct from previous attacks, result in distortions which do obey a small Lp norm constraint, and serve as unforeseen attacks for the ImageNet-UA attack suite.
does not provide an exhaustive guarantee over all conceivable attacks, it evaluates over diverse unforeseen test distribution similar to those used successfully other studies distributional shift Raajuurarr et al. 2018: Hendrycks & Dietterich) 2019; Recht et al. 2019). ImageNet-UA works for ImageNet models and can be easily used with our code available at htthtpss//gtthhhi com/
anon anon-submise www.sssumisssss.. www.:onnn-0002222 22200/ano--- anon- submi s sinoo---- 2020
Designing ImageNet-UA requires new attacks that are strong and varied, since real-world attacks diverse in structure. To meet this challenge, contribute four novel and diverse adversarial attacks which are easily optimized. Our new attacks produce distortions with occlusions, spatial similarity, and simulated weather, all of which are absent in previous attacks. Performing well on ImageNet-UA thus demonstrates that a defense generalizes to a diverse set of distortions distinct from the commonly used Loo L2-
With ImageNet-UA, we show weaknesses in existing evaluation practices and defenses through a study of attacks against 48 models adversarially trained ImageNet-100, a 100-class subset of ImageNet. While most adversarial robustness evaluations use only Loo attacks, ImageNet-UA reveals that models with high L attack robustness can remain susceptible to other attacks. Thus, Loo evaluations are narrow measure of robustness, even though much of the literature treats this evaluation as comprehensive (Madry al., 2017 Qian Wegman 2019 Schott al., 2019 Zhang et al., 2019). We address this deficiency by using the novel attacks in ImageNet-UA to evaluate robustness to a more diverse set unforeseen attacks. Our results demonstrate that L adversarial training. the current state-of-the-arr defense, has limited generalization to unforeseen adversaries, and is not casily improved by training against more attacks. This adds to the evidence that achieving robustness against few train-time attacks is insufficient to impart robustness unforeseen test-time attacks (Jacobsen et al. 2019 Jordan et al., 2019 Tramèr Boneh 2019).
In summary, propose the framework ImageNet-UA measure robustmess diverse set of attacks, made possible by our four new adversarial attacks, Since existing defenses scale poorly multiple attacks (Jordan et al. 2019: Tramer & Boneh 2019), finding defense techniques which generalize to unforeseen attacks is crucial to create robust models. We suggest ImageNet-UA as way to measure progress towards this goal.
RELATED WORK
Adversarial robustness is notoriously difficult to correctly evaluate (Papernot et al. 2017: Athalye et al. 2018a). To that end, Carlini et al. 2019a) provide extensive guidance for sound adversarial robustness evaluation. By measuring attack success rates across several distortion sizes and using broader threat model with diverse differentiable attacks, ImageNet-Ul has several of their recom- mendations built-in, while greatly expanding the of attacks previous work evaluation.
2

---

Under review as a conference paper at ICLR 2021
Randomly Initialized JPEG
Adversarially Optimized JPEG
Randomly Initialized Fog
Adversarially Optimized Fog
Otter (100.0%)
Basketball (100.0%)
Otter (100.0%)
Titi Monkey (100.0%)
Randomly Initialized Snow
Adversarially Optimized Snow
Randomly Initialized Gabor
Adversarially
Optimized Gabor
Otter (100.0%)
Loafer (98.0%)
Otter (100.0%)
Zebra (100.0%)
Figure 2: Randomly sampled distortions and adversarially optimized distortions from our new at- tacks, targeted to the target class in red. Stochastic average-case versions of our attacks affect classifiers minimally, while adversarial versions are optimized to reveal high-confidence errors. The snowflakes in Snow decrease in intensity after optimization, demonstrating that lighter adversarial snowflakes are more effective than heavy random snowfall at uncovering model weaknesses.
We are only aware of a few prior works which evaluate on unforeseen attacks in specific limited circumstances, [Wu et al. (2020) evaluate against thysically-realizabl attacks from Evtimov et al. (2017) and [Sharif al. (2016), though this limits the threat model occlusion attacks on narrow datasets. Outside of vision, Pierazzi et al. (2020) proposes constraining attacks by more diverse set of problem-space constraints in diverse domains such as text and malware or source code generation; however, even in this framework, analytically enumerating all such constraints is impossible.
Within vision, prior attacks outside the Lp threat model exist, but they lack the general applicability and fast optimization of ours. Song al.) (2018) and Qiu et al.] (2019) attack using variational autoencoders and StarGANs, respectively, resulting in weaker attacks which require simple image distributions suitable for VAEs and GANs. |Engstrom et al. (2017) apply Euclidean transformations determined by brute-force search. Zhao et al. (2019) use perceptual color distances to align human perception and L2 perturbations. [Evtimoy al. (207)) and (Sharif al. (2016) attack stop signs and face-recognition systems with carefully placed patches or modified eyeglass frames, requiring physical object creation and applying only to specific image types.
3 NEW ATTACKS FOR A BROADER THREAT MODEL
There are few diverse, easily optimizable, plug-and-play adversarial attacks in the current literature; outside of Elastic (Xiao al. 2018), most are Lp attacks such Loo (Goodfellow et al.) (2014), L (Szegedy et al. 2013] Carlini & Wagner 2017). L1 (Chen et al. 2018). We rectify this deficiency with four novel adversarial attacks: JPEG, Fog, Snow. and Gabor. Our attacks are differentiable and fast, while optimizing over enough parameters to be strong. We show example adversarial images Figure D and compare stochastic and adversarial distortions in Figure 2
Our novel attacks provide range of test-time adversaries visually and semantically distinct from Loo and Le attacks. Namely, they cause distortions with large Loo and La norm, but result in images that are perceptually close to the original. These attacks are intended as unforeseen attacks not used during training, allowing them to evaluate whether a defense can generalize from Loo or L2 to more varied set of distortions than current evaluations. Though attacks not exhaustive, performing well against them already demonstrates robustness to occlusion, spatial similarity, and simulated weather, which are absent from previous evaluations.

---

Under review as a conference paper at ICLR 2021
Our attacks create an adversarial image x' from a clean image with true label y. Let model f map images to a softmax distribution, and let {f)) y) be the cross-entropy loss. Given a target class y' y, our attacks attempt to find valid image x' such that (1) the attacked image x' is obtained by applying a distortion (of size controlled by parameter e) to 2; and (2) the loss {(((z') s/) is minimized, An unforeseen adversarial attack is white- or black-box adversarial attack unknown to the defense designer which does not change the true label of x according to an oracle or human.
3.1 FOUR NEW UNFORESEEN ATTACKS
JPEG. JPEG applies perturbations in JPEG-encoded space of compressed images rather than raw pixel space. More precisely, JPEG compression is linear transform JPEG which applies color- space conversion, the discrete cosine transform, and then quantization. Our JPEG attack imposes the -constrint
[JPEG JPEG(x')ll S
on the attacked image x'. We optimize = JP')(a') under this constraint to find an adversarial perturbation in the resulting frequency space. The perturbed frequency coefficients are quantized, and we then apply a right-inverse of JPEG to obtain the attacked image x' in pixel space. We use ideas from Shin & Song (2017) to make this differentiable. The resulting attack is conspicuously distinct from Lp attacks.
Fog. Fog simulates worst-case weather conditions. Robustness to adverse weather is safety crit- ical priority for autonomous vehicles, and Figure 2 shows Fog provides a more rigorous stress-test than stochastic fog (Hendrycks & Dietterich) 2019). Fog creates adversarial fog-like occlusions by adversarially optimizing parameters in the diamond-square algorithm (Fournier et al. 1982) typi- cally used render stochastic fog effects.
This algorithm starts with random perturbations to the four corner pixels of the image. At step I, it iteratively perturbs pixels at the centers of squares and diamonds formed by those pixels perturbed step t- 1. The perturbation of a step t pixel is the average of the neighboring step t-pt-1 perturbations plus parameter value which we adversarially optimize. We continue this process until all pixels have been perturbed; the outcome is fog-like distortion to the original image.
Snow simulates snowfall with occlusions of randomly located small image regions rep- resenting snowflakes. Because the distortions caused by snowflakes are not differentiable in their locations, we instead place occlusions representing snowflakes at randomly chosen locations and orientations and adversarially optimize their intensities. This choice results in fast, differentiable, and strong attack. Compared to synthetic stochastic snow (Hendrycks & Dietterich, 2019), our ad- versarial snow is faster and includes snowflakes at differing angles. Figure shows adversarial snow exposes model weaknesses more effectively than the easier stochastic, average-case snow.
Snow.
Gabor. Gabor spatially occludes the image with visually diverse Gabor noise Lagae et al. 2009)
Gabor noise is form of band-limited anisotropic procedural noise which convolves a parameter mask with Gabor kernel which is a product of a Gaussian kernel and harmonic kernel. We choose the Gabor kernel randomly and adversarially optimize the parameters of the mask starting from a sparse initialization. We apply spectral variance normalization (Co et al. 2019) the resulting distortion and add it to the input image to create the attack.
3.2 IMPROVING EXISTING ATTACKS
Elastic modifies the attack of Xiao et al. 2018); it warps the image by distortions x' Flow(x, V), where V
224] + o is a vector field on pixel space, and Flow sets the value of pixel (i,j) to the bilinearly interpolated original value at (i.j) + (i. j). We construct V by smoothing a vector field W by a Gaussian kernel (size 25 x 25, R 3 for 224 x 224 image) and optimize W under (il < s for all j.j. The resulting attack is suitable for large-scale images.
The other three attacks are L1, L2, Lox attacks, but we improve the L1 attack. For Loo and L2 constraints, we use descent and projection to the L and Lg balls (Madry et al. 2017). Projection is difficult for L1, and previous L, attacks rely on computationally intensive methods for it (Chen et al. 2018 Tramer & Boneh 2019). We replace PGD with the Frank-Wolfe algorithm (Frank & Wolfe 1956). which
projected gradient descent (PGD), which applies gradient
4

---

Under review as a conference paper at ICLR 2021
Accuracy
Accuracy
50
50-
25
0
0
2000
4000
0
10
15
L2 distortion size
Elastic distortion size
(a) L2 vs. L2-training
(b) Elastic vs. Tog-training
Figure 3: Accuracies of Lo and Elastic attacks at different distortion sizes against ResNet-50 model adversarially trained against Le at e 9600 on ImageNet- 100. At small distortion sizes, the model appears defend well against Elastic, but large distortion sizes reveal that robustness does not transfer from L2 to Elastic.
optimizes linear function instead of projecting each step (pseudocode in Appendix D This makes our L1 attack more principled than previous implementations.
4 ImageNet-UA: MEASURING ROBUSTNESS TO UNFORESEEN ATTACKS
We propose the framework ImageNet-UA and its CIFAR-10 analogue CIFAR-10-UA to measure and summarize model robustness while fulfilling the following desiderata: (1) defenses should be evaluated against a broad threat model through a diverse set attacks, (2) defenses should exhibit generalization to attacks not exactly identical to train-time attacks, and (3) the range of distortion sizes used for an attack must be wide enough to avoid misleading conclusions caused by overly weak strong versions of that attack (Figure 3)
The ImageNet-UA evaluation framework aggregates robustness information into a single measure, the mean Unforeseen Adversarial Robustness (mUAR). The mUAR is average six different attacks of the Unforeseen Adversarial Robustness (UAR), a metric which assesses the robustness of defense against a specific attack by using wide range of distortion sizes. UAR is normalized using a measure of attack strength, the ATA, which we now define.
Ad Adversarial Training Accuracy (ATA). The Adversarial Training Accuracy ATA(A, estimates the strength of an attack A against adversarial training (Madry et al. 2017). one of the strongest known defense methods. For distortion size E, it is the best adversarial test accuracy against A achieved by adversarial training against A. We allow a possibly different distortion size s' during training, since this can improves accuracy, and we choose fixed architecture for each dataset.
For ImageNet-100, we choose ResNer-50 for the architecture, and for CIFAR-10 we choose ResNet- 56. When evaluating defense with architecture other than ResNet-50 ResNet-56, we recommend using ATA values computed with these architectures to enable consistent comparison. To estimate ATA(A, E) in practice, we evaluate models adversarially trained against distortion size e for in large range (we describe this range at this section's 's end).
Single Attack. The UAR, building block for the mUAR, averages model's robustness to a single attack over six distortion sizes E chosen for each attack (we describe the selection procedure at the end of this section). It is defined as
UAR: Robustness Against
UAR(4) (4) = 100 x Σe Acc(A, Ex, M)
DE ATA(A,
where Acc(A, Eu, M) is the accuracy Acc(A, εx, M) of model M after attack A at distortion size εx. The normalization in (T) makes attacks of different strengths more commensurable in a stable way. We give values of ATA(A, and Eg for our attacks on ImageNet- 100 and CIFAR-I0 in Tables 4 and 55) (Appendix B). allowing computation of UAR of defense against a single attack with six adversarial evaluations and no adversarial training.
mUAR: Mean Unforeseen Attack Robustness. We summarize a defense's performance on ImageNeetUUA with the mean Unforeseen Attack Robustness (mUAR), average of UAR scores
5

---

Under review as a conference paper at ICLR 2021
Defense Robustness Under Different Attacks
Performance Defenses Adversarially Trained Against Loo
22
0
10
80
LAR(
Adversorialyy Trained Deferse
L.o
42
15
14
49
20
37
35
57
60
62
56
43
31
47
40
JPEG
65
70
54
32
40
19
31
52
20
Elastic
Fog
28
43
Standard
==1
==2 == 4 ==
16
32
Snow
13
15
1
39
37
98
60
Training
Figure 5: (A)( and mUAR for -tained models at different distortion sizes. Increasing distortion size in Lx training improves UAR(L but hurts the mUAR, suggesting models heavily fit Toc. at the cost of general- ization.
Gabor
JPEG
Elastic
Fos
Snow
Gobor
Adversarial
Attack
Figure 4: UAR for adv trained de- fenses (row) against attacks (col) on ImageNet- 100. Defenses from Lop to Gabor were trained with e = 32, 4.8k, 612k, 2, 16, 8192, 8, and 1.6k.
for the L1, Elastic, JPEG, Fog, Snow, and Gabor attacks:
mUAR 6 UUU_://LUUURAAAAAA---RRRRSAAAAAA
Our measure mUAR estimates robustness to a broad threat model containing six unforeseen attacks six distortion sizes each, meaning high mUAR requires generalization several held-out attacks. In particular, cannot be achieved by the common practice of engineering defenses to a single attack, which Figure shows does not necessarily provide robustness to different attacks.
Our four novel attacks play a crucial role in mUAR by allowing us to estimate robustness to a sufficiently large of adversarial attacks. As is customary when studying train-test mismatches and distributional shift, we advise against adversarially training with these six attacks when evaluating ImageNet-UA to preserve the validity of mUAR, though we encourage training with other attacks.
Distortion Sizes. We explain the e' values used to estimate ATA and the choice of E , used to define UAR. This calibration of distortion sizes adjusts for the fact (Figure 3 that adversarial robustness against an attack may vary drastically with distortion size. Further, the relation between distortion size and attack strength varies between attacks, so many too few Ey values in certain range may cause an attack to appear artificially strong or weak according to UAR.
We choose distortion sizes between Emin and émax as follows. The minimum distortion size m is the largest for which the adversarial accuracy of an adversarially trained model at distortion size e is comparable to that of a model trained and evaluated on unattacked data (for ImageNet- within 3 of 87). The maximum distortion size Emax is the smallest which either reduces adversarial accuracy of an adversarially trained model distortion size & below 25 or yields images confusing humans (adversarial accuracy can remain non-zero in this case).
As is typical in recent work on adversarial examples (Athalye et al.) 2018b Evtimov et al. 2017 Dong al. 2019 Qin et al. 2019), our attacks can be perceptible at large distortion sizes. We make this choice to reflect perceptibility of attacks in real world threat models per per[eem et al. (2018).
For ATA, we evaluate against models adversarially trained with s' increasing geometrically from Emin to Emax by factors of 2. We then choose Ek as follows: We compute ATA at E increasing geometrically from Emin to max by factors of 2 and take the size-6 subset whose ATA values have minimum & diddistance to the ATA values of the Los attack in Table 4 (Appendix B.I. For example, compute ATAS at the 10 values = 6.25. ... 3200. Viewing size-6 subsets of the ATAs as vectors with decreasing coordinates, we select E for Gabor corresponding to the vector with minimum E distancce to the ATA vector for Loc.
for Gabor, (Emin Emax) (6.25. 3200),

---

Under review as a conference paper at ICLR 2021
Table Clean Accuracy, UAR, and mUAR scores for models adv trained against Lx and Lg attacks. Lie training, the most popular defense, provides less robustness than L2 training. Comparing the highest mUAR achieved individual UAR values in Figure indicates large robustness gap.
Clean Accuracy
Loc
L2
mUAR
Clean Accuracy
Loo
L
mUAR
Normal Training
86.7
7.3
17.2
14.0
Normal Training
86.7
7.3
17.2
14.0
Lo
86.2
46.4
54.2
30.7
Le 150
86.6
38.0 49,4
27.1
Loe E
85.5
59.8 64.4
36.9
L2 300
85.9
49.7 60.1
33.3
Loe E
83.9
72.1 73.6
42.3
Ly E 600
84.7
61.9 71.6
40.0
Loo E 8
79.8
82.6
72.0
42.2
L2 = 1200
82.3
72.9 82.0
46.8
Loo 16
74.5
89.1
60.0
37.5
L2 2400
76.8
79.6 88.5
50.7
Loo E 32
70.8
88.1
41.9
31.8
L 1800
68.3
80.4
87.7
50.5
Table Clean Accuracy, UAR, and mUAR scores for models jointly trained against (Too, T2) Joint training does not provide much additional robustness.
Clean Accuracy
Li00
L22
mUAR
Loe L2 300
86.1
50.3
60.2
33.6
Tox 2, L2 & 600
85.1
62.8
72.5
41.0
Los = 4, L2 E 1200
81.3
72.9
81.2
46.9
Lis = 8, L2 E 2400
76.5
80.0
87.3
50.8
Lio 16, La 4800
68.4
81.5
87.9
50.9
NEW INSIGHTS FROM ImageNet-UA
We use ImageNet-UA assess existing methods for adversarial defense and evaluation. First, ImageNet-UA reveals that Loe trained defenses fail to generalize to different attacks, indicating substantial weakness in current Loo adversarial robustness evaluation. We establish baseline for ImageNet-UA using Le adversarial training which is difficult to improve upon by adversarial training alone. Finally, we show adveraaararlllyy trained models can still improve robustness on ImageNet-Uu UA over standard models and suggest this as direction for further inquiry.
5.1 EXPERIMENTAL SETUP
We adversarially train 48 models against the 8 attacks from Section 3 and evaluate against targeted attacks. We use the CIFAR-10 and ImageNet- 100 datasets for ImageNet-UA and CIFAR-10-UA, ImageNet-1001 100 is 100-class subset of ImageNet-IK (Deng et 2009) containing every tenth class by WordNer ID order; we use a subset of ImageNet-IK due to the high compute cost of adversarial training. We use ResNet-56 for CIFAR-10 and ResNet-50 from torchvis sison for ImageNet-100 (He et al., 2016). We provide training hyperparameters in Appendix [A]
To adversarially train against attack A, at each mini-batch we select a uniform random (incorrect) target class for each training image. For maximum distortion size e, we apply targeted attack A the current model with distortion size Uniform(O,c E) and take SGD step using only the attacked images. Randomly scaling e' improves performance against smaller distortions.
We train on 10-step attacks attacks other than Elastic, where we use 30 steps due to harder optimization. For T.p. JPEG, and Elastic, we use step size e/ s//Stepp /Steps; for Fog, Gabor, and Snow, use step size /0.001 /steps because the latent space is independent of e. These choices have optimal rates for non-smooth convex functions (Nemirovski & Yudin 1978 1983). We evaluate on 200-step targeted attacks with uniform random (incorrect) target, using more steps for evaluation than training per best practices (Carlini et al. 2019b).
Figure summarizes ImageNet-100 results. Full results for ImageNet-100 and CIFAR-I0 are in Appendix and robustness checks to random seed and attack iterations are in Appendix
5.2 ImageNet- UA REVEALS WEAKNESSESS L TRAINING AND TESTING
We ImageNet-UA reveal weaknesses in the common practices of Lao robustness evaluation and Lx: adversarial training. We compute the mUAR and (AR(L for models trained against the Tax attack with distortion size and show results in Figure 5 For small e 4, mUAR and

---

Under review as a conference paper at ICLR 2021
Table 3: Non-adversarial defenses can noticeably improve ImageNet-UA performance. ResNeXt-
101 (32x8d) + WSL is trained on approximately billion images Mahajan et al. (2018). Stylized
ImageNet is trained on modification of ImageNet using style transfer Geirhos transfer[Geirhos et al. (2099) Patch Gaussian augments using Gaussian distortions small portions of the image Lopes et al. (2019).
AugMix mixes simple random augmentations of the image Hendrycks et al. (2020). These results
suggest that ImageNet-UA performance may be achieved through non-adversaria defenses.
Clean Acc.
Teo
1,2
L1
Elastic
JPEG
Fog
Snow
Gabor
mUAR
SqueezeNet
84.1
5.2
11.2
14.9
25.9
1.9
20.1
9.8
4.4
12.8
ResNeXt-1101 (32x8d)
95.9
2.5
5.5
20.7
26.5
1.8
14.1
12.4
5.3
13.4
ResNeXt-101 (32x8d) WSL
97.1
3.0
5.7
28.3
29.4
1.9
26.2
20.3
8.0
19.0
ResNet-18
91.6
8.2
22.6
1.8
20.3
4.2
ResNet-50
94.2
2.7
6.6
20.1
24.9
1.8
15.8
11.9
4.9
13.2
ResNet-50 Stylized ImageNet
94.6
2.9
7.4
22.8
26.0
1.8
16.2
12.5
8.1
14.6
ResNet-50 Patch Gaussian
93.6
4.5
10.9
27.4
28.2
1.8
23.9
10.5
5.2
16.2
ResNet-50 AugMix
95.1
13.4
34.3
38.8
1.8
28.6
24.7
11.1
23.2
s.) increase together with e. For larger E 2 (AR(L()) continues to increase with E, but the mUAR decreases, fact which is not apparent from Lxx evaluation.
The decrease mUAR while (AR(L() increases suggests that Loo adversarial training begins to heavily fit Lo distortions at the expense of generalization at larger distortion sizes. Thus, while it the most commonly used defense procedure, Los. training may not lead improvements on other attacks or to real-world robustness.
Worse, Tox evaluation against Lox adversarial training higher distortions indicates higher robust- ness. In contrast, mUAR reveals that Lox adversarial training higher distortions in fact hurts robustness against more diverse set of attacks. Thus, Los evaluation gives a misleading picture of robustness. This is particularly important because Lee evaluation is the most ubiquitous measure of robustness in deep learning (Goodfellow et al. 2014 Madry et al., 2017 Xie et al. 2018).
5.3 LIMITS OF ADVERSARIAL TRAINING FOR ImageNet-UA
We establish a baseline on ImageNet- UA using L2 adversarial training but show a significant perfor- mance gap even for more sophisticated existing adversarial training methods. To do so, we evaluate several adversarial training methods on ImageNet- UA and show results in Table
Our results show that L2 trained models outperform L.c trained models and have significantly im- proved absolute performance, increasing mUAR from 14.0 to 50.7 compared an undefended model. The individual UAR values in Figure 7 (Appendix I improve substantially against all attacks other than Fog, including several (Elastic, Gabor, Snow) of extremely different nature to L2-
This result suggests pushing adversarial training further by training against multiple attacks simul- tancously via joint adversarial training (Jordan et al.] 2019) Tramer & Bonch 2019) detailed in Appendix C Table 2 shows that, despite using twice the compute of L2 training, (Loo, L2) joint training only improves the m UAR from 50.7 to 50.9. We thus recommend L2 training as a baseline for ImageNet-UA, though there is substantial room for improvement compared to the highest UARs against individual attacks in Figure 4 which are all above 80 and often above 90.
5.4 ImageNet-UA ROBUSTNESS THROUGH NON-ADVERSARIAL DEFENSES
We find that methods can improve robustness unforeseen attacks without adversarial training. Ta- ble shows mUR for SqueezeNet (landola et al. (017). ResNeXts (Xie et al. (0066), and ResNets. For ImageNet-1K K models, we mask 900 logits to predict ImageNet- classes.
A popular defense against average case distortions (Hendrycks & Dietterich, 2019) is Stylized Im- ageNet (Geirhos et al. 2019), which modifies training images using image style transfer hopes of making networks rely less textural features. Table shows it provides some improvement on ImageNet-UA. More recently, Lopes et al. (2019) propose to train against Gaussian noise applied to small image patches, improving the mUAR by 3% over the ResNet-50 baseline. The second largest mUAR improvement comes from training ResNeXt on approximately 1 billion images (Maha
jan al. 2018). This three orders of magnitude increase in training data yields 5.4% mUAR
8

---

Under review as a conference paper at ICLR 2021
increase over vanilla ResNeXt baseline. Finally, Hendrrykks et al. (2020) create AugMix, which randomly mixes stochastically generated augmentations. Although AugMix did not use random nor adversarial noise, it improves robustness unforeseen attacks by 10%.
These results imply that defenses not relying on adversarial examples can improve ImageNet-UA performance. They indicate that training more data only somewhat increases robustness on ImageNet- UA, unlike many other robustness benchmarks (Hendrycks & Dietterich, [2019] Hendrycks al., 2019) where more data helps tremendously (Orhan 2019). While models with lower clean accuracy (e.g.. SqueezeNet and ResNet- 8) have higher UAR(Loo) and (AR(L2) than many other models, there is no clear difference mUAR. Last, these non- non-ad adversarial defenses have minimal cost to accuracy on clean examples, unlike adversarial defenses. Much remains to explore, and we hope non-adversaraal defenses will promising avenue toward adversarial robustness.
6 CONCLUSION
This work proposes a framework ImageNet-UA to evaluate robustness a defense against unforeseen attacks. Because existing adversarial defense techniques do not scale to multiple attacks, developing models which can defend against attacks not seen at train-time essential for robustness. Our results using ImageNet-UA show that the common practice of Loc training and evaluation fails to achieve or measure this broader form of robustness. As a result, it can provide misleading sense of robustness. By incorporating our novel and strong adversarial attacks, ImageNet-UA enables evaluation on the diverse held-out attacks necessary to measure progress towards robustness more broadly.
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprini arXiv: arXiv: 1802.80420, 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adver- sarial examples. In Jennifer Dy and Andreas Krause (eds.), Proceedings the 35th Interna- tional Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Re- search, pp. 284-293, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018b. PMLR. URL http:://proceedinggs mlr. prwss /BO/ 20t00//thlllelleee 1el8b. html
Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. CoRR, abs/1712.096655. 2017. URL htt.thttp///aaxtto www./ogg////ooeoooo/ooooeeeeeree abs/ 09665
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEFF, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian G Goodfellow, and Aleksander Madry. On evaluating adversarial robustness: Princi- ples of rigorous evaluations. 2019a.
Nicholas Carlini, Anish Athalye, Nicolas Papemot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, lan J. Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. CoRR, htss/192.006755 2019b. URL BtR.http://arst www.riiiioor//.....e www.oo/gg/b.aaa..oooooo 902. 06705
Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. EAD: Elastic-net attacks to deep neural networks via adversarial examples. In Thirly-second AAAI conference on artificial intelligence, 2018
Kenneth T. Co, Luis MuñoozGGonzález, and Emil C. Lupu. Sensitivity of deep convolutional net- works to Gabor noise. CoRR, abs/1906.03455, 2019. URL http:://rxxv..ogg/
1906. 03455)
Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via ran- domized smoothing. CoRR, abs//992..0291s 2019. URL UtL.ttrp://axivvoor////tt
02918

---

Under review as a conference paper at ICLR 2021
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scald hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, IEEE, 2009.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invaric attacks. In Proceedings of the IEEE Computer Society Confer- ence on Computer Vision and Pattern Recognition, 2019.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and translation suffice: Fooling CNNs with simple transformations. arXiv preprint arXiv:1712.02779, 2017.
Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Xiaodong Song. Robust physical-world attacks on deep learning models. 2017.
Alain Fournier, Don Fussell, and Loren Carpenter. renderiggg of stochastic models. Com- mum. ACM, 25(6)3371-384 June 1982. ISSN 0001-0782. doi: 10. 10.1145/3853333355533 URL http://dcz: acm. wwt//o../.........oo....
358553
Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2):95-110, 1956.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. tragenet-trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019.
wttp://wwwwweeeeeeureeeeeeeeee Ltteee/Eooouunnii--yy999999999 URL httpap/ss..ppeuueeeeeeeee/e///aa
Justin Gilmer, Ryan P. Adams, Ian J. Goodfellow, David Andersen, and George E. Dahl. Motivating the rules of the game for adversarial example research. ArXiv, /5//1807.06732 2018.
Ian J. Goodfellow, A research agenda: Dynamic models to defend against correlated attacks. ArXiv, abs/1903.06222 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577. 2014.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv preprint 170.::770.006611 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07174 2019.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference Learning Representations (ICLR), 2020.
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: AlexNet-level accuracy with 50x fewer parameters and <bmb model size. ArXiv, abs/1602.07360, 2017.
ploiting excessive invariance caused by boundddnded adversarial robustness, 2019. Lm--Herik Jacobsen, Jens Behrmannn, Nicholas Carlini, Florian Tramr, and Nicolas Papernot. Ex-
Matt Jordan, Naren Manoj, Surbhi Goel, and Alexandros G. Dimakis. Quantifying perceptual dis- tortion of adversarial examples. arXiv e-prints, art. arv: 192vv1900.088266 Feb 2019.
10

---

Under review as a conference paper at ICLR 2021
Ares Lagae, Sylvain Lefebvre, George Drettakis, and Philip Dutré. Procedural noise using sparse Gabor convolution. ACM Trans. Craph., 28(3):54:1 July 2009. ISSN 0730-0301. doi: 10. (145/1531322.1531333 URLhatts /doi. acm. org/ 10..:..14451111 1ttt://0...111/5133333...3333311
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin Dogus Cubuk. Improving ro- bustness without sacrificing accuracy with patch gaussian augmentation. ArXiv, abss/990.026111 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 185-201, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01216-8.
Arkadi Nemirovski and D Yudin. On Cezari's convergence of the steepest descent method for approximating saddle point convex-concave functions. In Soviet Math. Dokl, volume 19, pp- 258-269, 1978.
Arkadi Nemirovski and D Yudin. Problem Complexity and Method Efficiency in Optimization. Intersci. Ser. Discrete Math. Wiley, New York, 1983.
A. Emin Orhan. Robustness properties of Facebook's ResNeXt WSL models. ArXiv, abs/1907.07640, 2019.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-519. ACM, 2017.
Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro. Intriguing prop- erties adversarial ml attacks in the problem space. In 2020 IEEE Symposium on Secu- rity and Privacy (SP), pp. 1308- 1325. IEEE Computer Society, 2020. doi: 10. 10.1109/SP40000. 2020.00073, URL 2020.00072
Haifeng Qian and Mark N. Wegman. Le-nonexpansive neural networks. In Internationnal Conference on Learning Representations (ICLR), 2019. URL hhttps://pennreiiew.nrt wttp:/rwwwwiiieeeeefoeeeeee
id=ByxGSsR9FD
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization, 2019.
Haonan Qiu, Chaowei Xiao, Lei Yang, Xinchen Yan, Honglak Lee, and Bo Li. Semanticadv: Gener- ating adversarial examples via attribute-conditinaal image editing. ArXiv, abs/1906.07927, 2019.
Edward Raff, Jared Sylvester, Steven Forsyth, and Mark McLean. Barrage of random transforms for adversarially robust defense. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6528-6537, 2019.
Rajpurkar, R. Jia, and P. Liang. Know what you don't know: Unanswerable questions for SQuAD. In Association for Computational Linguistics (ACL), 2018
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
L. Schott, J. Rauber, W. Brendel, and M. Bethge. Towards the first adversarially robust neural network model on MNIST. May 2019. URL http:::/ http:::://axiv.ooo http///awwv.ooppp//////i///e//// 0919C. pdf

---

Under review as a conference paper at ICLR 2021
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize crime: Real and stealthy attacks on state-of-the-ar face recognition. In Proceedings of the 23rd ACM SIGSAC Conference Computer and Communications Security, 2016.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. A general framework for adversarial examples with objectives. ACM Transactions on Privacy and Security (TOPS), 22(3): 1-30, 2019.
Richard Shin and Dawn Song. JPEG-resistant adversarial images. In NIPS 2017 Workshop on Machine Learning and Computer Security, 2017.
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In NewrlPS, 2018.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations.
arXiv e-prints, art. arXiv: 1904. 1904.190000 Apr 2019.
Florian Tramer, Pascal Dupré, Gili Rusak, Giancarlo Pellegrino, and Dan Boneh. Ad-varaiial:: Defeating perceptual ad- blocking. CoRR, abs/18111.03194, 2018. URLaattp:/arxix orq/ abs 18 11881118 03194
Tong Wu, Liang Tong, and Yevgeniy Vorobeychik. Defending against physically realizable attacks on image classification. In International Conference on Learning Representations, 2020. URL
https://openreview net forum?ideeei forummid=+4LxsssEKCC
Chaowei Xiao, Jun- Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially trans- formed adversarial examples. arXiv preprint arXiv: arXiv:1801.02612 2018
Cihang Xie, Yuxin Wu, Laurens der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. arXiv preprint arXiv: /0/2.034//33 2018.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987-5995, 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing. Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learming, volume 97 Proceedings of Machine Learning Research, pp. 7472-7482, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http:://proceddingg.. press/ w97/zhang19p.. html
Zhengyu Zhao, Zhuoran Liu, and Marisa Larson. Towards large yet imperceptible adversarial image perturbations with perceptual color distance, ArXiv, abs//991..02400 2019.
12

---

Under review as a conference paper at ICLR 2021
A TRAINING HYPERPARAMETERS
For ImageNet- 100, we trained on machines with NVIDIA VI00 GPUs using standard data aug- mentation (He et al. 2016). Following best practices for multi-GPU training (Goyal et al. 2017) we ran synchronized SGD for 90 epochs with batch size x2x8 and a learning rate schedule with "warm-up" epochs and decay epochs 30, 60, and 80 by factor of 10. Initial learning rate after warm-up was 0.1, momentum was 0.9, and weight decay was 10-4 For CIFAR-10 10, we trained on single NVIDIA V100 GPU for 200 epochs with batch size 32, initial learning 0.1, momentum 0.9, and weight decay 10-1 We decayed the learning rate at epochs 100 and 150.
B CALIBRATION OF ImageNet-UA AND CIFAR-10-UA
B. CALIBRATION FOR ImageNet-UA
Calibrated distortion sizes and ATA values are in Table
B.2 CALIBRATION FOR CIFAR-10-UA
The ε calibration procedure for CIFAR-IO was similar to that used for ImageNet-100. We started with small Emi values and increased & geometrically with ratio until adversarial accuracy of an udversarially trained model dropped below 40. Note that this threshold is higher for CIFAR 1- than ImageNet- 100 because there are fewer classes. The resulting ATA values for CIFAR-10 10 are shown in Table 5
C JOINT ADVERSARIAL TRAINING
Our joint adversarial training procedure for attacks and A' is follows. At each training step, we compute the attacked image under both A and A' and backpropagate with respect to gradients induced by the image with greater loss. This corresponds to the "max" loss of Tramer Bonch 2019). We train ResNet-50 models for (Loc, L2), (Loo, L1), and (Loo, Elastic) on ImageNet-100,
Table shows training against (Loc, L1) is worse than training against Lı at the same distortion size and performs particularly poorly at large distortion sizes. Table 7 shows joint training against
Table 4: Calibrated distortion sizes and ATA values for different distortion types on ImageNet-100 100.
Attack
81
E2
E3
E4
E5
E6
ATA
ATA₂
ATA3
ATA
ATA;
ATAg
Loc
2
4
8
16
32
84.6
82.1
76.2
66.9
40.1
12.9
L2
150
300
600
1200
2400
4800
85.0
83.5
79.6
72.6
59.1
19.9
L1
9562.5
19125
76500
153000
306000
612000
84.4
82.7
76.3
68.9
56.4
36.1
Elastic
0.25
0.5
2
4
16
85.9
83.2
78.1
75.6
57.0
22.5
JPEG
0.062
0.125
0.250
0.500
85.0
83.2
79.3
72.8
34.8
1.1
Fog
128
256
512
2048
4096
8192
85.8
83.8
79.0
68.4
67.9
64.7
Snow
0.0625
0.125
0.25
84.0
81.1
77.7
65.6
59.5
41.2
Gabor
6.25
12.5
25
400
800
1600
84.0
79.8
79.8
66.2
44.7
14.6
Table 5: Calibrated distortion sizes and ATA values for ResNet-56 on CIFAR-10
Attack
E1
E2
E3
E4
Es
E6
ATA
ATA
ATA
ATA,
ATAS
ATAs
Loe
2
4
8
16
32
91.0
87.8
81.6
71.3
46.5
23.1
L2
40
80
160
320
640
2560
90.1
86.4
79.6
67.3
49.9
17.3
L1
195
390
780
1560
6240
24960
92.2
90.0
83.2
73.8
47.4
35.3
JPEG
0.03125
0.0625
0.125
0.25
0.5
89.7
87.0
83.1
78.6
69.7
35.4
Elastic
0.125
0.25
0.5
2
87.4
81.3
72.1
58.2
45.4
27.8
13

---

Under review as a conference paper at ICLR 2021
Table UAR scores for L1 -trained models and (Le , L Lij-onntly trained models. At each distortion size, Li-trinning performs better than joint training.
UAR Az
UAR
Loo L1 E 76300
48
66
Loo 4. L1 153000
51
72
Loo & 8, L1 E 306000
62
L1 76500
50
70
L1 E = 153000
81
L1 E 306000
59
87
Table UAR scores for - and Elastic-trained models and (L Elastic)-ooinll trained models. No jointly trained model matches a Elastic-traines model on UAR vs. Elastic.
UAR
UAR
Loo 4, Elastic e 2
68
Loo 8, Elastic 4
35
65
Loe 16, Elastic & =
69
43
Elastic 2
37
68
Elastic = 4
36
81
Elastic 8
31
91
(Los, Blastic) also performs poorly, never matching the UAR score of training against Elastic at moderate distortion size 2).
D THE FRANK- TRANK-WOLFE ALGORITHM
We chose the Frank-Wolfe algorithm for optimizing the In attack, Projected Gradient Descent would require projecting onto a truncated L1 ball, which is a complicated operation. In contrast, Frank-Wolfe only requires optimizing linear functions g x over a truncated Lı ball; this can be done by sorting coordinates by the magnitude of and moving the top & coordinates to the boundary of their range (with & chosen by binary search). This is detailed Algorithm
E FULL EVALUATION RESULTS
E. FULL EVALUATION RESULTS AND ANALYSIS FOR IMAGENET-100
We show the full results of all adversarial attacks against all adversarial defenses for ImageNet-100 in Figure 6 These results also include LP-JPEG and L2-JPEG attacks, which modifications of the JPEG attack applying -opconnttaintss in the compressed JPEG space instead of Loo constraints. Full UAR scores provided for ImageNet-100 in Figure 7
E.2 FULL EVALUATION RESULTS AND ANALYSIS FOR CIFAR-10
We experienced difficulty training the L2 and L1 attacks distortion sizes greater than those shown and have omitted those runs, which we believe may be related to the small size of CIFAR- 10 images. Full UAR values for CIFAR-10 are shown Figure
We show the results of adversarial attacks and defenses for CIFAR-10 in Figure g
F ROBUSTNESS OF OUR RESULTS
F.1 REPLICATION
We replicated our results for the first three rows of Figure 6 with different random seeds to see the variation in our results. As shown in Figure 10 deviations in results minor.
14

---

Under review as a conference paper at ICLR 2021
022
4030
227 71 1000
Figure 6: Accuracy of adversarial attack (column) against adversarially trained model (row) on ImageNet- 100.
12
Watee
12
10 n 5 6
2
0222
6102222
28
13
(2
4031
4
10 20 15 1x
24
15 34 18
10.0 24
15:01 H
02
10:0
:
HEB 6 14
24
2
11 12 53
80
0261
2M
13:2
YE41
HE
11
33
1
L
3
1
33
3
6
33
03:44
1n 1728
15 60
07
14 02
1 583323
64 28
0373-000 6613
022
6
12
21
2
0.0
29
0
0
5.0
0.0
0
0
1
14 1
03
17:00
25
19-01
12
12 14
10:00
03.0 15
13:0 1724
78
11 14
73 41
19/15 24
12:222241
24
04.0 00
11 Mi 1
11 29
015
130011 11
640
01
000 M
21 14
21
9322
01
N6G 28
000 5 5
2 2
00
00 28
5
14 14
11.00
01
6822
60
10
14
1851 1i55--11 108 10
1s
10
22
2y 2000
37-0000 680
705311
Liper- - 20012
L i8
iilear -
1,900 120
RO -20
A - 1054
330072
18
rec 1008
for 1014
rog 0511
Gaser 621
doter 121
GADEE 20
GEOI -00
GenT -10
tom 20s 213
Gater
15

---

Under review as a conference paper at ICLR 2021
Algorithm Pseudocode for the Frank- algorithm for the Lı attack.
Input: function f, initial input x E [0. 1]d, Ia radius ps number of steps T.
2: Output: approximate maximizer I of f over the truncated I1 ball B1 (p:x)) n [0, 11 centered
at æ.
3:
4: (:))) <- Randomlnit(z)) {Random initialization)
5: for = 1..... T do
6:
g f V[(x!(-11) {Obtain gradient)
7:
for k 1,... do
8:
S/ +- index of the coordinate of g by with h largest norm
9:
end for
10: Sp t {51 $
12:
(Compute move to boundary of [0, 1] for each coordinate.]
13: for ......... d do
14:
if g: > 0 then
15:
b, + 2;
16:
else
17:
b. f- -;;;
18:
end if
19:
end for
Mr + Eies, [b:] [Compute L--petuurbttion moving & largest coordinates.] }
21:
k²* max(x M₁ 0} {Choose largest & satisfying L1 constraint.)
22:
(Compute è maximizing g x over the L, ball.}
24:
for = 1...... do if E Sy- then
26:
2, -- + b;
27:
else if = 81+41 then
28:
2; < I, (p- M& ) (g.)(g.)
29:
else
30:
2; < I;
31:
end if
32:
end for
33: x! +- (1 } (1-1) + Le (Average & with previous iterates) }
end for
35: F t (T)
F.2 CONVERGENCE
We replicated the results in Figure 6 with 50 instead of 200 steps to see how the results changed based on the number of steps in the attack. As shown in Figure 11 the deviations are minor.
16

---

Under review as a conference paper at ICLR 2021
Namral Traiing
17
22
16
DO
Nornal Trainia
3D
Nora Thining
37
22
46
Fx
17
40
Pa 150
70
9001
31
40
43
72
42
Za 200
Ty 19135
49
5x
73
48
Za 600
Ey 20000
Ax m
Ag 1200
f 150000
Ag 9100
339991
55
75
613000
2y.PEG
Ebetic
Fbotic
Tog 128
12
54
41
Gaber 6.20
Snow e D40225 15
32
Tog 339
22
36
50
13
Gaber 12.5
11
Snow 0,125
22
20
23
86
28
Tng 5:3
18
88
56
58
10
22
Golberr 30
22
Shate 2:15
15
16
24
50
50
Fig r 2014
26
20
31
Sahoor 100
30
22
Skoow
24
28
Fog 4305
24
00
29
34
Galoer 804
20
Shoo
24
25
Poge 5202
2e
43
Gabar 100
Show
15
28
37
60
Fig
Colle
Black
Fig
Colice
Figure UAR scores for adv. trained defenses (rows) against distortion types (columns) for ImageNet-100.
17

---

Under review as a conference paper at ICLR 2021
Recesarial eccuracy
1.0
0.2
0.0
33
45
Figure 8: Accuracy of adversaial attack (column) against adversarially trained model (row) on CIFAR-I0 100
26 38
1 52
E0
1
aee o 1 Taa da Woaaa 122
15
35 11
2
38
52
65
83 53 12
78
38 14
00 80 79 79
28
20
46 27 11
21
24
68
73
77
77
63
09 01 62
58
63
71
85 65
38
67 90
92 91 RP n
BE 06 BE B4
85 02
01
50
02
09
01
62
m
09
00 90
06
TO 79
9 01 OF
9O 07
06
50 52
54
50 90 89
0S 05
D3 06
74
85
m
9 90 90
09
51
83
Br
52
W.
91
92
3 2
20 63
T4
77 73
29
56
72
77 5
04 06 05 01 74
aa as a
38
56
69
0
Se 09 04 56
80
0
91 08 88 39
m
1
89 4F
5
22
01
96 58
08
71
00
85.00 m 79
09 70 17
58
28
76
58
85
85
T7
79
07
68
72
11
62
T9
01 19
53 17
06 06 06 85 05 04
79
00 20 00 79 79
25
89 24 22
07
06
35
89 79
90 90 m
07
85
05
71 32
49 24
S 30
50
88
05
82
06
00
7e
76
78
07
06 86 96 BC
83
52.01
30
53
02
60
96
01 01
04
05
00 50 90
99.99
03 93992900
92
01
mn n0 m
91
96
00 29 06
01 S 90
98
51511
02
02 52
0 30
06 56
79 74
do de
77 22
22 91
50
28
92
01
93 52
91 S1
OB 20
06 84
60 59
92
92
02
12
53
0
13
02
AF
01
03
02
03
24
Lojee ee a 2..70 Pade 1 a-d ant
n
14
0
70 35
0
0
0
0
0
14
25
50
155
12
56
20
79
37
02
14 69 52
1
28
63
75
77
0
20
78
9
55
82 72
77
20:28 0
23 3
57 24
53
30
68
26
45
78
07 83
83
76
27
82
86
52 5
59
80
82
56
78
26
19
73
50
00
04
76 76
12
10
20 6
24
49
00.03
07
50
65
05
79
19
50 4
83 48
50
05
07
03
88
05
05
66 03
64 46
s3 37
36
24
42
36
36
33
35
55
02 20
M 22
M 20
10
F4
35
72
52 15
82 88 3
76 7
7
70 25
88
88
86
06
86
78
M
80
M
di 19 3e0 t
03
12
36
6
57
67
0
67 26
25
50
67
12
42
75 65
13
72
62
23
66
58
18
78
83
92 n0
39
92 07 63 11
TO
77
T
83
89
59
82
57
50
86
10
48
R5 49
43
45
78
93 01 53
01 09 55
85
66
R
RC 03 92
DU S 53
53 83
88 08
11 55 92
01 9
92 91
00 90
00 RC
60 0R
75
01 92
92 07
69 78
VO 02
45
85 05
62 82
75 75
92 90
90 85
R7 87
B6 85
50 DE
23
92
02
53
63
07
92
09
45
02
02
02
00
89
13
11
M8 X
63 <s
28
0
12
49
63
73
65
13
60
05
55
22
44
n6 m 15
68 28
85 38
13
33
64
13
62
66
67
79 46
65
59
20
09 69 21
87 79 38
85 79 50
17
79
86
08
87
02
02
m1 mB
67
m
06
07
86
05
79
73
23
m8
74
62
31
T6
13
48
59
m1
04
69
79
79
77
744 23
38
05 58
85 69
n0 m
02
02
78 67
64
Sa 53
51
4y-180 10
-67
-01
-69
-93
-9
-03
-1
53
L -1
Bbatk 25 -11
Basii E 02
Le
1,
1,122
1,x-40
1,1-00
4. 320
12-1220
4,5-2560
52-5220
115-115
L,E-300 300
LLE-
4,c-1560
4,c-6200
by byc-12400
byc-0920 49920
L. 1ire--00330 003225
L -1
Ly 00.12
L L.I c-22 e225
kopt 16
4,955 x- 32
LoPEE -M
4,y00 -28 128
L.pEG - 255
L.PEo - 512
LyjPEG 1024
Elatic 6212
Elsiie E 4-4
Elaiic
18

---

Under review as a conference paper at ICLR 2021
Normal Training
17
16
48
5
25
Normal Training
17
16
48
5
25
3
Normal Training
17
16
48
5
25
Lx 49
89
31
37
53 53
74
32
38
4 195
36
38
70
34
Lx
390
Lsc
160
84
780
Lox c 72
77
50
39
43
320
81
88
64
46
18 15
80
10
Loe $ =
69
75
78
55
40
51
L2 640
84
86
88
70
51
52
L1 6240
17
13
10
36
Fou 32
72
76
58
48
45
1.2 2500
87
85
86
78
71
66
T1 490120
49
47
61
47
51
29
L. JPEG e 0.031225
48
43
61
51
42
17
Li-JPEG = 2
39 37
62
32
41
12
Elastic 0.125 40
37
63
19
24
41
Ioc JPEG 0.0625-
54
50
66
63
49
21
I-JJEG 8
49 47
68
54
51
Elastic
41
38
65
23
25
JPEG 0.125
89
58
JPEG
58
76
66
Elastic
40
65
28
27
Tw JPEG 0.25
LP-J 256
Elastic :
Ie-JPEG
73
Ly-JEGG 512
76
87
Elastic :
-JEG c
64
71
94
35
T.IPPG
68
66
75
88
Elastic 45
31
37 27
1m
V8
Low Tow-JPEO JPEG
1A-JPEG
Elastic
Joe-PPES
La
La IAJPEE Elaatic
19
Low- JPEG
La Ly-JPEG
Elastic
Figure 9: UAR scores on CIFAR-10. Displayed UAR scores are multiplied by 100 for clarity.
Ao 07
Goid 85
85
88
81 72
84
88
Au 04
Lo 2
24
73 67 34
24 M 19
61 48 16
25
24
67
(adversrial trairin))
29
150
82
88
EX
latiooe (suesiaipy
300
86
84
21
86 68
82 19
84
88
83
44
000
12 2000
27
00
27
70
RT
71
00 22
77
77
25
07
00
59
61
69
67
69
59
50
69
8002.44
78
82
E4
26
19135
88
77
0
84
84
33
700
84
83 20
35250.1
L. 153000
80
81
21
76
50
81
80
4. 530000
70
00
7B
0.0
lo D00
co 048244
20
7000
PPEG oo as
8000
JPEG 02
133
La JPB surd5
le
280
Battle 0.25
Elatt i
LedREG
IPEG
JPED
Lo- IPEG
Clatt
Eartio
Battee
Elsit
In
L.-FEE FFEG
le JPSS
Attack (evaluation)
Figure 10: Replica of the first three block rows of Figure in results are minor.
with with different
random seeds. Deviations
19

---

Under review as a conference paper at ICLR 2021
1.0
Figure 11: Replica of Figure 6 with 50 steps instead of 200 evaluation time, Deviations in results are minor.
16 120
a
1
1
6
035322
574021
14482
12
16
323300
a FaN 2 a y
5
10 28
2072
15
11
203
14.0 21 5
18 1824 D3
H4 23
16. 100
1T00 28
10 6
77
15 D
12
11.00
1000
12
1
:
:
3
00
17
1181 (1 28
2014
53
2812
11.0 0.
1222
14.30
0518
3
3
33
5313
10 12 1458
12 0322
00 020
10 16
to 5216
4
220324
to 18 0044
1 22521
IT 122223
662244
5
12 2014
66 2014
12 2212
5355 19
14.2
13
14
67
10
15
68
2.1
2230
53 1 3
63
0
0
0
0
0
0
1
0
11
0
0
15:0 08 18
16:2 64
(1 28
10:3 0
25
03
19:00
12:0222
+4.00
17
00 68
00.22.0 29
11
21
19 11
19773 24
71 X5
12.0 20
10 11
0007 79
M
24
N60
1358 of
2435
6 0
09248
0778-1 22
8
12.08 km
01
25
05 2015
11
00
01-00 LB
00
63
68
12
18
Asc ti 3
1 1s in
30000
Site 10000
12000
L -1
L. 312
L. p
Loft// 298
1o86 (- 500
1:00 - aa
1 1104 18M4
J 3878
hg/ -00
fog 301
Qater 123
100
Geter 1800
GaM e 3300
0 6GS
towr -1 115
100 2
65
frow ia
20


Reviews:
[{"review_id": "hKTPvaEqPss", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update**: Thanks to the authors for addressing my comments and releasing the source code. Code is well-structured and easy to follow. It can be definitely used as a supplement for the robustness evaluation. However, judging by the implementation, the rain, snow, and fog attacks are too simplistic. From the paper, it is not clear how well these attacks approximate the respective type of perturbations in real-world scenarios. Therefore, the proposed set of 6 attacks is more the author's heuristic than something that can be used for the evaluation of computer vision systems, e.g. autonomous vehicles. Given all that, I decrease my score from 5 to 4.\n\n###### Summary\nThe paper proposes a novel benchmark for the evaluation of the model's robustness against unforeseen adversaries. The authors suggested using 6 types of adversaries: JPEG, FOG, SNOW, and GABOR attack, L1, ELASTIC. Fully-differentiable variants of the above attacks are based on the previous work. The authors introduced the novel $l_1$-norm attack, which uses the Frank-Wolf algorithm to satisfy $l_1$-perturbation constraint. The authors introduced two metrics to compare defenses: robustness against a single unforeseen attack and mean unforeseen robustness. Experiments with adversarially trained models and non-adversarially trained models align with the previous work in the area.\n\n###### Reasons for score: \nI vote for a weak accept. The paper studies an important problem of the robustness of unforeseen adversaries. Introducing novel benchmarks beyond $l_{p}$-norm robustness is an important problem. The paper is clearly written and easy to follow. The evaluation results are extensive. However, implementation details of some attacks (SNOW, FOG, GABOR) are missing. JPEG attack is based on the previous work. It is not clear if the authors plan to release the benchmark suite to the reviewers. The most similar work in Hendrycks et al. 2019 uses a more diverse set of attacks.\n\n###### Concerns:\n- Novel differentiable attacks are based on previous work, e.g. differentiable JPEG and FOG. The algorithm and implementation details for SNOW and GABOR attacks are not provided. It is not clear how to reimplement these attacks.\n- The paper is the most similar to Hendrycks & Dietterich 2019. The main difference is the use of differentiable attacks. However, previous work in Hendrycks uses a larger set of corruptions. This work includes only 5 corruptions from Hendrycks et al. 2019. Do the authors plan to include other types of attacks for the Imagenet-UA benchmark?\n- It is not clear from the main text if the authors plan to release the benchmark source code during the reviewing process.\n  \n###### Minor comments:\n- Comment that Elastic $l_1$ attack is based on heuristics might be incorrect. Elastic $l_1$ attack uses a proximity operator, a principal way to minimize $l_1$-norm.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gtTxBXqdNoZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe authors propose ImageNet-UA, a framework to evaluate the\neffectiveness of test-time unforeseen adversarial ML attacks. In this\nparticular context, the authors observe that attackers are not limited\nto induce minimal Lp-norm perturbations and therefore it is important to\nunderstand how robust existing defenses are against attacks that diverge\nfrom these constraints while still being realizable. Within their\nframework, the authors propose four novel adversarial attacks to\nconsolidate our understanding of test-time adversarial ML attacks.\n\n## Strengths\n\n+  Interesting intuition that test-time adversarial attacks are not all\nnecessarily driven by minimal perturbations\n+  Attack-evaluation framework available to the community\n\n## Weaknesses\n\n-  Recent work explores the need for having a unified theoretical\n   framework to reason about realizable attacks\n-  Actual benefit of the approach\n\n## Comments\n\nThis is an interesting paper that explores an avenue that is often\nneglected in the context of test-time adversarial ML attacks. There is a\ntendency on focusing on attacks that minimize the perturbation in a\ngiven Lp-norm, when in fact there are several application domains for\nwhich this constraint does not necessarily matter. Here, in the context\nof realizable or problem-space attacks, one is required to reformulate\nthe problem, which may include other constraints attacks must satisfy.\n\nAs the authors admit, ImageNet-UA is not exhaustive although it tries to\nevaluate the robustness of defenses over diverse unforeseen test\n(adversarial) distributions. Under this premise, I somehow fail to see\nthe real benefit of such a framework, unless the authors show such\nadditional attacks comprise widely-used transformations that should\ntherefore become part of the threat model of any adversarial work in the\nproblem space. (In addition, most of the attacks seem to be a marginal\nimprovement of existing ones.) Conversely, a theoretical reformulation of the\nproblem-space would have been perhaps more useful. That would have\nallowed one to reason about properties or set of transformations one\ncould or should consider, along with other constraints, as outlined in\n[1]. \n\nI also wonder what's the rationale behind the mUAR metric. It is\ndefinitely useful to have a way to capture the robustness of models\nagainst unforeseen attacks, but I wonder whether an average as opposed\nto, e.g., a geometric mean, would provide useful insights. In a way, the\naverage would consider all the attacks equally challenging where, in\nreality, this might be the case. (I do actually like the use of mUAR and UAR tho,\nas outlined in Section 5.2).\n\n## Additional comments\n\nIn Section 4, the authors explain how the distortion size \\epsilon_max\nis chosen: \"[...] the smallest ε which either reduces adversarial\naccuracy of an adversarially trained model at distortion size ε below 25\nor yields images confusing humans\". How is this verified? Do the authors\nrely on a user study? Does this only rely on Gilmer et al. (2018)?\n\n[1] https://s2lab.kcl.ac.uk/projects/intriguing/ (IEEE S&P 2020)", "extra_scores": {}, "extra_text": {}}, {"review_id": "_oYvtEQmlvm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents the existence of several adversarial attacks that have meaningful visual concepts. Based on these attacks, the authors further develop a method to measure the robustness of a network or an adversarial defense. \n\nDespite being conceptually interesting, this paper is working on the thing that many studies have done. The related work section is not on the point, in my opinion, the core problem (''unforeseen adversaries'') of this work should be ''adversarial attacks that are designed to break the commonly employed defenses'', but not ''adversarial attacks that are visually meaningful, and occasionally break the commonly employed defenses''. Therefore, the related work should focus on these studies that propose measures of robustness, and the experimental analysis should focus on doing a comparison of these measures, telling why the readers should adopt the ImageNet-UA and mUAR as their baseline, but not others.\n\nOn the other hand, the analysis of these newly developed is not sufficient, and the experimental results can not support the claims convincingly. For instance, in Table 3, the JPEG attack acquires the worst performance and the data augmentation methods do not help in all scenarios. Meanwhile, the Fog and Snow attacks are less effective than the $L_\\inf$ and $L_2$ and benefit from more data. This implies they have different dynamics thus should not be used for mUAR homogeneously.\n\nAnyway, I suggest the authors carefully revise the motivation and experimental analysis.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mxm69kh1WQl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes four novel efficient adversarial attack methods beyond Lp threat models. Together with other two existing attack methods, these six attack methods combine as a framework to evaluate robustness of defenses against unforeseen attacks. In this framework, the novel measure is normalized with the performance of adversarial training. The experiments show that the Linf adversarially trained model may not lead to improvement of robustness against other threat models. It is expected that the framework could help test model robustness.\n\n## Advantages\n\n- Evaluating model robustness against unforeseen adversaries is an important problem.\n- The four proposed adversarial attacks are differentiable and easy to use.\n\n## Disadvantages\n\n- The proposed measure mUAR is quite heuristic, e.g., the choice of the epsilon parameter.\n- As a baseline framework, it is hard to say that the six adversarial attacks are sufficient to evaluate robustness against unforeseen adversaries. As a result, the claims seem not to be very convincing.\n- In general, the result that adversarial training is not robust to unforeseen attacks is not surprising and has be been discussed a lot in literature.  It would be good to evaluate more existing defenses in order to derive more novel, interesting and inspiring insights.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, a defense method against test-time adversarial ML attacks is proposed. Unfortunately, it is not clear whether the proposed method is practically useful or not, because the types of attacks assumed in this paper are too simple and heuristic. Also, the position of the proposed method in the vast amount of existing research on adversarial attacks is not clear. Although the proposed method is conceptually interesting, no evidence is provided that the proposed method is significantly superior to existing approaches.
Decision: Reject